[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:45:50 dist-test-slave-cx89 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:45:50 dist-test-slave-cx89 krb5kdc[399](info): set up 2 sockets
Jun 20 14:45:50 dist-test-slave-cx89 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 14:45:51.041584 392 mini_kdc.cc:121] Time spent starting KDC: real 2.576s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 14:45:51.206918 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.165s user 0.001s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 14:45:51.358728 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.152s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 14:45:51.529345 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.171s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/test-user.keytab.
W20260620 14:45:52.499948 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.970s user 0.000s sys 0.001s
Jun 20 14:45:52 dist-test-slave-cx89 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:45:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:45:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260620 14:45:52.691000 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.171s user 0.002s sys 0.001s
I20260620 14:45:52.691468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32779
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:45:52.774165 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:45:52.774435 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:45:52.774469 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:45:52.776037 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:45:52.776090 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:45:52.776110 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:45:52.776126 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:45:52.776151 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:45:52.778191 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:45:52.778417 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:45:52.778604 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:45:52.783471 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:52.783479 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:52.783617 412 server_base.cc:1034] running on GCE node
W20260620 14:45:52.783481 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:52.783953 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:45:52.784241 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:45:52.785379 412 hybrid_clock.cc:648] HybridClock initialized: now 1781966752785359 us; error 34 us; skew 500 ppm
Jun 20 14:45:52 dist-test-slave-cx89 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:45:52.787684 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:45:52.788111 412 webserver.cc:460] Webserver started at http://127.0.98.62:41441/ using document root <none> and password file <none>
I20260620 14:45:52.788300 412 fs_manager.cc:359] Metadata directory not provided
I20260620 14:45:52.788347 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:45:52.788467 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:45:52.789487 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "cd7f3c66333b45c5bc3cbb603e7b4866"
format_stamp: "Formatted at 2026-06-20 14:45:52 on dist-test-slave-cx89"
server_key: "01ea651957b81d1e8d33f02b2ceb60a2"
server_key_iv: "dee2e9356f7a303f123b608d52c978a2"
server_key_version: "encryptionkey@0"
I20260620 14:45:52.789824 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "cd7f3c66333b45c5bc3cbb603e7b4866"
format_stamp: "Formatted at 2026-06-20 14:45:52 on dist-test-slave-cx89"
server_key: "01ea651957b81d1e8d33f02b2ceb60a2"
server_key_iv: "dee2e9356f7a303f123b608d52c978a2"
server_key_version: "encryptionkey@0"
I20260620 14:45:52.791062 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:45:52.791764 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:52.791900 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:52.791980 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "cd7f3c66333b45c5bc3cbb603e7b4866"
format_stamp: "Formatted at 2026-06-20 14:45:52 on dist-test-slave-cx89"
server_key: "01ea651957b81d1e8d33f02b2ceb60a2"
server_key_iv: "dee2e9356f7a303f123b608d52c978a2"
server_key_version: "encryptionkey@0"
I20260620 14:45:52.792038 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:45:52.821482 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:45:52.823693 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:45:52.823858 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:45:52.827811 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32779
I20260620 14:45:52.827848 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32779 every 8 connection(s)
I20260620 14:45:52.828186 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:45:52.829061 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:52.831564 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866: Bootstrap starting.
I20260620 14:45:52.832114 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:52.832332 480 log.cc:824] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866: Log is configured to *not* fsync() on all Append() calls
I20260620 14:45:52.832942 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866: No bootstrap required, opened a new log
I20260620 14:45:52.834458 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7f3c66333b45c5bc3cbb603e7b4866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } }
I20260620 14:45:52.834597 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:52.834619 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7f3c66333b45c5bc3cbb603e7b4866, State: Initialized, Role: FOLLOWER
I20260620 14:45:52.834733 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7f3c66333b45c5bc3cbb603e7b4866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } }
I20260620 14:45:52.834784 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:45:52.834802 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:45:52.834831 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:52.835311 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7f3c66333b45c5bc3cbb603e7b4866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } }
I20260620 14:45:52.835390 480 leader_election.cc:302] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd7f3c66333b45c5bc3cbb603e7b4866; no voters:
I20260620 14:45:52.835553 480 leader_election.cc:288] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:45:52.835592 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:45:52.835747 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866 [term 1 LEADER]: Becoming Leader. State: Replica: cd7f3c66333b45c5bc3cbb603e7b4866, State: Running, Role: LEADER
I20260620 14:45:52.835788 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:45:52.835832 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7f3c66333b45c5bc3cbb603e7b4866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } }
I20260620 14:45:52.836205 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd7f3c66333b45c5bc3cbb603e7b4866. Latest consensus state: current_term: 1 leader_uuid: "cd7f3c66333b45c5bc3cbb603e7b4866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7f3c66333b45c5bc3cbb603e7b4866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } } }
I20260620 14:45:52.836277 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 412
I20260620 14:45:52.836349 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866 [sys.catalog]: This master's current role is: LEADER
I20260620 14:45:52.836333 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd7f3c66333b45c5bc3cbb603e7b4866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7f3c66333b45c5bc3cbb603e7b4866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32779 } } }
I20260620 14:45:52.836383 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:45:52.836401 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866 [sys.catalog]: This master's current role is: LEADER
I20260620 14:45:52.836553 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:45:52.837198 392 external_mini_cluster.cc:1428] Setting key 2bc04f337d923734a719da0106c14a88
I20260620 14:45:52.837203 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:45:52.838732 492 catalog_manager.cc:1347] Generated new cluster ID: 8e2df5a5037c4139b97c09000be8061d
I20260620 14:45:52.838783 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:45:52.846312 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:45:52.847033 492 catalog_manager.cc:1504] Loading token signing keys...
Jun 20 14:45:52 dist-test-slave-cx89 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:45:52.849651 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd7f3c66333b45c5bc3cbb603e7b4866: Generated new TSK 0
I20260620 14:45:52.849835 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:45:52.854429 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:52.838081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35706 (local address 127.0.98.62:32779)
0620 14:45:52.839628 (+ 1547us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:52.839637 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:52.839670 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:45:52.840357 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:52.840363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:52.840426 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:52.840537 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:52.841976 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:52.842927 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:52.843938 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:52.844362 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:52.844458 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:52.849126 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:52.849131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:52.849131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:52.849145 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:52.852192 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:52.852640 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:52.852644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:52.852644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:52.852736 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:52.852994 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:52.852995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:52.852996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:52.853239 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:52.853386 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:52.854048 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 14:45:52.854199 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1505,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:45:52.880580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32779
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:45:52.965408 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:45:52.965636 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:45:52.965660 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:45:52.966822 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:45:52.966862 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:45:52.966889 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:45:52.968732 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32779
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:45:52.968982 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:45:52.969204 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:45:52.973943 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:52.973953 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:52.973955 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:52.974181 506 server_base.cc:1034] running on GCE node
I20260620 14:45:52.974383 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:45:52.974611 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:45:52.975757 506 hybrid_clock.cc:648] HybridClock initialized: now 1781966752975744 us; error 28 us; skew 500 ppm
Jun 20 14:45:52 dist-test-slave-cx89 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:45:52.978631 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:45:52.979031 506 webserver.cc:460] Webserver started at http://127.0.98.1:43699/ using document root <none> and password file <none>
I20260620 14:45:52.979246 506 fs_manager.cc:359] Metadata directory not provided
I20260620 14:45:52.979288 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:45:52.979444 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:45:52.980474 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "548dcb9ea82c46f48ff2e15ccf3adb55"
format_stamp: "Formatted at 2026-06-20 14:45:52 on dist-test-slave-cx89"
server_key: "26afa6728f6ebfcebdb3d5728f4005fd"
server_key_iv: "0f6736fc97792704f9f4896390d73417"
server_key_version: "encryptionkey@0"
I20260620 14:45:52.980792 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "548dcb9ea82c46f48ff2e15ccf3adb55"
format_stamp: "Formatted at 2026-06-20 14:45:52 on dist-test-slave-cx89"
server_key: "26afa6728f6ebfcebdb3d5728f4005fd"
server_key_iv: "0f6736fc97792704f9f4896390d73417"
server_key_version: "encryptionkey@0"
I20260620 14:45:52.981981 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:45:52.982640 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:52.982793 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:45:52.982872 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "548dcb9ea82c46f48ff2e15ccf3adb55"
format_stamp: "Formatted at 2026-06-20 14:45:52 on dist-test-slave-cx89"
server_key: "26afa6728f6ebfcebdb3d5728f4005fd"
server_key_iv: "0f6736fc97792704f9f4896390d73417"
server_key_version: "encryptionkey@0"
I20260620 14:45:52.982940 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:45:52.995718 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:45:52.997416 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:45:52.997553 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:45:52.997750 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:45:52.998032 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:45:52.998064 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:52.998092 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:45:52.998109 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:53.003345 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36529
I20260620 14:45:53.003402 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36529 every 8 connection(s)
I20260620 14:45:53.003706 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:45:53.004863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 506
I20260620 14:45:53.004930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:45:53.005062 392 external_mini_cluster.cc:1428] Setting key 0c858c58a54495e49799ff58a56a2fd7
Jun 20 14:45:53 dist-test-slave-cx89 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:45:53.012965 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:53.004297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36717 (local address 127.0.98.62:32779)
0620 14:45:53.004338 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:53.004340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:53.004550 (+ 210us) server_negotiation.cc:407] Connection header received
0620 14:45:53.005088 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:53.005090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:53.005108 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:53.005154 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:53.006164 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:53.006661 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:53.007662 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:53.007940 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:53.008006 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:53.010299 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:53.010300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:53.010300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:53.010307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:53.011574 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:53.012007 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:53.012008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:53.012009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:53.012056 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:53.012306 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:53.012307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:53.012307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:53.012401 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:53.012446 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:53.012805 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 14:45:53.012895 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:45:53.013078 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:53.004291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32779 (local address 127.0.98.1:36717)
0620 14:45:53.004435 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:53.004470 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:53.004992 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:53.005236 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:53.005241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:53.005872 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:53.006073 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:53.006079 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:53.006740 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:53.006741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:53.007588 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:53.007591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:53.008014 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:53.008015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:53.008128 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:53.008572 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:53.008580 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:53.010209 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:53.011645 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:53.011652 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:53.011654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:53.011937 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:53.012115 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:53.012116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:53.012116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:53.012251 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:53.012460 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:53.012465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:53.012634 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 14:45:53.012749 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:45:53.012892 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260620 14:45:53.013598 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32779
I20260620 14:45:53.013700 635 heartbeater.cc:464] Registering TS with master...
I20260620 14:45:53.013875 635 heartbeater.cc:511] Master 127.0.98.62:32779 requested a full tablet report, sending...
I20260620 14:45:53.014392 444 ts_manager.cc:194] Registered new tserver with Master: 548dcb9ea82c46f48ff2e15ccf3adb55 (127.0.98.1:36529)
I20260620 14:45:53.015141 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:45:53.028388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32779
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:45:53.107443 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:45:53.107635 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:45:53.107656 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:45:53.108765 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:45:53.108805 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:45:53.108841 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:45:53.110227 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32779
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:45:53.110436 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:45:53.110637 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:45:53.115446 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:53.115476 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:53.115613 640 server_base.cc:1034] running on GCE node
W20260620 14:45:53.115473 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:53.115846 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:45:53.116083 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:45:53.117218 640 hybrid_clock.cc:648] HybridClock initialized: now 1781966753117208 us; error 32 us; skew 500 ppm
Jun 20 14:45:53 dist-test-slave-cx89 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:45:53.119503 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:45:53.119905 640 webserver.cc:460] Webserver started at http://127.0.98.2:38353/ using document root <none> and password file <none>
I20260620 14:45:53.120105 640 fs_manager.cc:359] Metadata directory not provided
I20260620 14:45:53.120152 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:45:53.120270 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:45:53.121238 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "f43e5c77679643a48ee3d56e5f2e9967"
format_stamp: "Formatted at 2026-06-20 14:45:53 on dist-test-slave-cx89"
server_key: "6c7436b1282dad60543020ae8aed362c"
server_key_iv: "058450812cbc08c22a1f285f3478f5b2"
server_key_version: "encryptionkey@0"
I20260620 14:45:53.121564 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f43e5c77679643a48ee3d56e5f2e9967"
format_stamp: "Formatted at 2026-06-20 14:45:53 on dist-test-slave-cx89"
server_key: "6c7436b1282dad60543020ae8aed362c"
server_key_iv: "058450812cbc08c22a1f285f3478f5b2"
server_key_version: "encryptionkey@0"
I20260620 14:45:53.122798 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:45:53.123445 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:53.123587 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:45:53.123664 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "f43e5c77679643a48ee3d56e5f2e9967"
format_stamp: "Formatted at 2026-06-20 14:45:53 on dist-test-slave-cx89"
server_key: "6c7436b1282dad60543020ae8aed362c"
server_key_iv: "058450812cbc08c22a1f285f3478f5b2"
server_key_version: "encryptionkey@0"
I20260620 14:45:53.123723 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:45:53.149222 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:45:53.151129 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:45:53.151265 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:45:53.151510 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:45:53.151799 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:45:53.151832 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:53.151862 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:45:53.151881 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:53.157433 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38875
I20260620 14:45:53.157495 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38875 every 8 connection(s)
I20260620 14:45:53.157773 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:45:53.162587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 640
I20260620 14:45:53.162664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:45:53.162822 392 external_mini_cluster.cc:1428] Setting key 465e1c9b0207874a7e1a0a84a0c71c06
Jun 20 14:45:53 dist-test-slave-cx89 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:45:53.166699 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:53.158495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44863 (local address 127.0.98.62:32779)
0620 14:45:53.158546 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:53.158548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:53.158719 (+ 171us) server_negotiation.cc:407] Connection header received
0620 14:45:53.159211 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:53.159212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:53.159225 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:53.159277 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:53.160197 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:53.160676 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:53.161634 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:53.161826 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:53.161881 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:53.164116 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:53.164118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:53.164118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:53.164124 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:53.165467 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:53.165835 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:53.165836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:53.165836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:53.165898 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:53.166108 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:53.166109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:53.166109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:53.166220 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:53.166276 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:53.166529 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 14:45:53.166610 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:45:53.166867 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:53.158463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32779 (local address 127.0.98.2:44863)
0620 14:45:53.158603 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:53.158641 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:53.159110 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:53.159349 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:53.159353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:53.159914 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:53.160100 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:53.160105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:53.160755 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:53.160756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:53.161549 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:53.161551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:53.161893 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:53.161894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:53.162010 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:53.162450 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:53.162458 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:53.164012 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:53.165542 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:53.165548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:53.165551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:53.165756 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:53.165951 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:53.165951 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:53.165952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:53.166051 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:53.166274 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:53.166281 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:53.166413 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 14:45:53.166502 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:45:53.166654 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260620 14:45:53.167245 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32779
I20260620 14:45:53.167348 769 heartbeater.cc:464] Registering TS with master...
I20260620 14:45:53.167541 769 heartbeater.cc:511] Master 127.0.98.62:32779 requested a full tablet report, sending...
I20260620 14:45:53.167918 444 ts_manager.cc:194] Registered new tserver with Master: f43e5c77679643a48ee3d56e5f2e9967 (127.0.98.2:38875)
I20260620 14:45:53.168417 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44863
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:45:53.186585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32779
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:45:53.266252 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:45:53.266461 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:45:53.266491 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:45:53.268870 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:45:53.268972 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:45:53.269023 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:45:53.271368 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32779
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:45:53.271827 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:45:53.272078 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:45:53.277839 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:53.277839 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:53.277941 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:53.278622 774 server_base.cc:1034] running on GCE node
I20260620 14:45:53.278841 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:45:53.279111 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:45:53.280274 774 hybrid_clock.cc:648] HybridClock initialized: now 1781966753280254 us; error 34 us; skew 500 ppm
Jun 20 14:45:53 dist-test-slave-cx89 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:45:53.283902 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:45:53.284492 774 webserver.cc:460] Webserver started at http://127.0.98.3:42513/ using document root <none> and password file <none>
I20260620 14:45:53.284762 774 fs_manager.cc:359] Metadata directory not provided
I20260620 14:45:53.284821 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:45:53.284970 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:45:53.286419 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "55ce35cf2aee411aabdb25d73b255939"
format_stamp: "Formatted at 2026-06-20 14:45:53 on dist-test-slave-cx89"
server_key: "00ea155bed15d49dd2321646bf3df6a0"
server_key_iv: "817e7a62a6c53898ae45d10a2c96250e"
server_key_version: "encryptionkey@0"
I20260620 14:45:53.286931 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "55ce35cf2aee411aabdb25d73b255939"
format_stamp: "Formatted at 2026-06-20 14:45:53 on dist-test-slave-cx89"
server_key: "00ea155bed15d49dd2321646bf3df6a0"
server_key_iv: "817e7a62a6c53898ae45d10a2c96250e"
server_key_version: "encryptionkey@0"
I20260620 14:45:53.288589 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:45:53.289389 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:53.289621 774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:45:53.289713 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "55ce35cf2aee411aabdb25d73b255939"
format_stamp: "Formatted at 2026-06-20 14:45:53 on dist-test-slave-cx89"
server_key: "00ea155bed15d49dd2321646bf3df6a0"
server_key_iv: "817e7a62a6c53898ae45d10a2c96250e"
server_key_version: "encryptionkey@0"
I20260620 14:45:53.289779 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:45:53.317632 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:45:53.319603 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:45:53.319711 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:45:53.319890 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:45:53.320122 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:45:53.320163 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:53.320183 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:45:53.320196 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:53.326205 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46287
I20260620 14:45:53.326253 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46287 every 8 connection(s)
I20260620 14:45:53.326548 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:45:53.331202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 774
I20260620 14:45:53.331290 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:45:53.331451 392 external_mini_cluster.cc:1428] Setting key 2ac03f71c73ffeb7f8183c6c9517dc8a
Jun 20 14:45:53 dist-test-slave-cx89 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:45:53.335356 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:53.327186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48987 (local address 127.0.98.62:32779)
0620 14:45:53.327231 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:53.327233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:53.327410 (+ 177us) server_negotiation.cc:407] Connection header received
0620 14:45:53.327921 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:53.327922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:53.327933 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:53.327979 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:53.328894 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:53.329387 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:53.330337 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:53.330521 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:53.330575 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:53.332840 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:53.332841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:53.332841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:53.332848 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:53.334095 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:53.334454 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:53.334455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:53.334456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:53.334511 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:53.334733 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:53.334734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:53.334734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:53.334854 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:53.334909 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:53.335189 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 14:45:53.335267 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:45:53.335552 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:53.327147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32779 (local address 127.0.98.3:48987)
0620 14:45:53.327298 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:53.327336 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:53.327830 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:53.328056 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:53.328061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:53.328609 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:53.328806 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:53.328810 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:53.329464 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:53.329465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:53.330259 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:53.330261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:53.330586 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:53.330587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:53.330710 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:53.331176 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:53.331184 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:53.332760 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:53.334167 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:53.334173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:53.334176 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:53.334390 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:53.334563 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:53.334564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:53.334564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:53.334677 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:53.334907 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:53.334912 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:53.335046 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 14:45:53.335164 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 14:45:53.335330 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
I20260620 14:45:53.335990 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32779
I20260620 14:45:53.336068 903 heartbeater.cc:464] Registering TS with master...
I20260620 14:45:53.336233 903 heartbeater.cc:511] Master 127.0.98.62:32779 requested a full tablet report, sending...
I20260620 14:45:53.336592 444 ts_manager.cc:194] Registered new tserver with Master: 55ce35cf2aee411aabdb25d73b255939 (127.0.98.3:46287)
I20260620 14:45:53.336972 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48987
I20260620 14:45:53.343526 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:45:53 dist-test-slave-cx89 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:45:53 dist-test-slave-cx89 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:45:53.365072 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:53.358182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52322 (local address 127.0.98.1:36529)
0620 14:45:53.358345 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:53.358350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:53.358360 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:45:53.358385 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:53.358387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:53.358400 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:53.358464 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:53.358911 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:53.359677 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:53.360352 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:53.360674 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:53.360751 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:53.362721 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:53.362728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:53.362729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:53.362736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:53.364204 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:53.364472 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:53.364475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:53.364475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:53.364535 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:53.364681 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:53.364682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:53.364682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:53.364793 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:53.364889 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:53.364927 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 14:45:53.364983 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
W20260620 14:45:53.365411 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52322
Jun 20 14:45:53 dist-test-slave-cx89 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:45:53 dist-test-slave-cx89 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:45:53.385280 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:53.378894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52326 (local address 127.0.98.1:36529)
0620 14:45:53.378945 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:53.378946 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:53.378990 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:45:53.379064 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:53.379065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:53.379077 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:53.379121 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:53.379783 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:53.380265 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:53.381000 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:53.381196 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:53.381253 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:53.383280 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:53.383281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:53.383281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:53.383291 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:53.384338 (+ 1047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:53.384617 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:53.384618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:53.384618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:53.384665 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:53.384834 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:53.384835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:53.384835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:53.384928 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:53.385033 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:53.385102 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:45:53.385180 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:45:53.385744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 506
I20260620 14:45:53.390446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 640
I20260620 14:45:53.395082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 774
I20260620 14:45:53.399044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 412
2026-06-20T14:45:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4957 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:45:53 dist-test-slave-cx89 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:45:53 dist-test-slave-cx89 krb5kdc[917](info): set up 2 sockets
Jun 20 14:45:53 dist-test-slave-cx89 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 14:45:54.354909 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:45:54 dist-test-slave-cx89 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:45:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:45:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:45:54.464377 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34091
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:45:54.541678 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:45:54.541887 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:45:54.541908 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:45:54.543036 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:45:54.543077 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:45:54.543090 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:45:54.543102 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:45:54.543111 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:45:54.544569 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:45:54.544793 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:45:54.544991 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:45:54.549507 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:54.549563 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:54.549507 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:54.549714 930 server_base.cc:1034] running on GCE node
I20260620 14:45:54.549990 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:45:54.550266 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:45:54.551405 930 hybrid_clock.cc:648] HybridClock initialized: now 1781966754551389 us; error 38 us; skew 500 ppm
Jun 20 14:45:54 dist-test-slave-cx89 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:45:54.553754 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:45:54.554201 930 webserver.cc:460] Webserver started at http://127.0.98.62:34499/ using document root <none> and password file <none>
I20260620 14:45:54.554401 930 fs_manager.cc:359] Metadata directory not provided
I20260620 14:45:54.554445 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:45:54.554546 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:45:54.555624 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "0166e555a4a348a5a9738f6245050ea1"
format_stamp: "Formatted at 2026-06-20 14:45:54 on dist-test-slave-cx89"
server_key: "556da1425e39eb3a82181cad49abbcf5"
server_key_iv: "b7dc36e68df38ac6a4b626a778539f73"
server_key_version: "encryptionkey@0"
I20260620 14:45:54.555949 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "0166e555a4a348a5a9738f6245050ea1"
format_stamp: "Formatted at 2026-06-20 14:45:54 on dist-test-slave-cx89"
server_key: "556da1425e39eb3a82181cad49abbcf5"
server_key_iv: "b7dc36e68df38ac6a4b626a778539f73"
server_key_version: "encryptionkey@0"
I20260620 14:45:54.557152 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:45:54.557830 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:54.558017 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:45:54.558102 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "0166e555a4a348a5a9738f6245050ea1"
format_stamp: "Formatted at 2026-06-20 14:45:54 on dist-test-slave-cx89"
server_key: "556da1425e39eb3a82181cad49abbcf5"
server_key_iv: "b7dc36e68df38ac6a4b626a778539f73"
server_key_version: "encryptionkey@0"
I20260620 14:45:54.558173 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:45:54.571393 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:45:54.573246 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:45:54.573365 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:45:54.577016 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34091
I20260620 14:45:54.577059 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34091 every 8 connection(s)
I20260620 14:45:54.577366 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:45:54.578141 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:54.578336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 930
I20260620 14:45:54.578408 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:45:54.578548 392 external_mini_cluster.cc:1428] Setting key 7f478b687413c110a8323687638196df
I20260620 14:45:54.580852 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1: Bootstrap starting.
I20260620 14:45:54.581512 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:54.581773 998 log.cc:824] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:45:54.582396 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1: No bootstrap required, opened a new log
I20260620 14:45:54.584259 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0166e555a4a348a5a9738f6245050ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34091 } }
I20260620 14:45:54.584426 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:54.584460 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0166e555a4a348a5a9738f6245050ea1, State: Initialized, Role: FOLLOWER
I20260620 14:45:54.584582 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0166e555a4a348a5a9738f6245050ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34091 } }
I20260620 14:45:54.584661 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:45:54.584695 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:45:54.584740 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 14:45:54 dist-test-slave-cx89 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:45:54.585242 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0166e555a4a348a5a9738f6245050ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34091 } }
I20260620 14:45:54.585328 998 leader_election.cc:302] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0166e555a4a348a5a9738f6245050ea1; no voters:
I20260620 14:45:54.585467 998 leader_election.cc:288] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:45:54.585518 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:45:54.585618 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1 [term 1 LEADER]: Becoming Leader. State: Replica: 0166e555a4a348a5a9738f6245050ea1, State: Running, Role: LEADER
I20260620 14:45:54.585695 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0166e555a4a348a5a9738f6245050ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34091 } }
I20260620 14:45:54.585722 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:45:54.586086 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0166e555a4a348a5a9738f6245050ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0166e555a4a348a5a9738f6245050ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34091 } } }
I20260620 14:45:54.586163 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:45:54.586967 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0166e555a4a348a5a9738f6245050ea1. Latest consensus state: current_term: 1 leader_uuid: "0166e555a4a348a5a9738f6245050ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0166e555a4a348a5a9738f6245050ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34091 } } }
I20260620 14:45:54.587054 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1 [sys.catalog]: This master's current role is: LEADER
W20260620 14:45:54.587598 1019 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:45:54.587652 1019 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:45:54.587697 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:45:54.587852 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:45:54.588959 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:54.579020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42230 (local address 127.0.98.62:34091)
0620 14:45:54.579207 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:54.579216 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:54.579248 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:45:54.579766 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:54.579774 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:54.579836 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:54.579983 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:54.580516 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:54.581777 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:54.582501 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:54.582912 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:54.583003 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:54.585247 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:54.585251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:54.585251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:54.585262 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:54.587165 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:54.587517 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:54.587520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:54.587520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:54.587618 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:54.587893 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:54.587894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:54.587895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:54.588184 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:54.588337 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:54.588637 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 14:45:54.588763 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260620 14:45:54.589306 1011 catalog_manager.cc:1347] Generated new cluster ID: f83b1b866592419bacc2192ae4a9e715
I20260620 14:45:54.589354 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:45:54.643962 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:45:54.644717 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:45:54.649019 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0166e555a4a348a5a9738f6245050ea1: Generated new TSK 0
I20260620 14:45:54.649202 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:45:54.678406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34091
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:45:54.755462 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:45:54.755657 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:45:54.755687 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:45:54.756801 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:45:54.756839 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:45:54.756873 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:45:54.758219 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34091
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:45:54.758452 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:45:54.758646 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:45:54.764159 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:54.764348 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:54.764166 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:54.764727 1024 server_base.cc:1034] running on GCE node
I20260620 14:45:54.764932 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:45:54.765164 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:45:54.766309 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781966754766293 us; error 33 us; skew 500 ppm
Jun 20 14:45:54 dist-test-slave-cx89 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:45:54.771122 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:45:54.771628 1024 webserver.cc:460] Webserver started at http://127.0.98.1:38149/ using document root <none> and password file <none>
I20260620 14:45:54.771847 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 14:45:54.771891 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:45:54.771994 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:45:54.773008 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "8f63de0b4612428290ea53de454a8967"
format_stamp: "Formatted at 2026-06-20 14:45:54 on dist-test-slave-cx89"
server_key: "926432ca057e2455cdffedf860a0fa8f"
server_key_iv: "64bf968521b32e0ca4db49666ca41d08"
server_key_version: "encryptionkey@0"
I20260620 14:45:54.773320 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8f63de0b4612428290ea53de454a8967"
format_stamp: "Formatted at 2026-06-20 14:45:54 on dist-test-slave-cx89"
server_key: "926432ca057e2455cdffedf860a0fa8f"
server_key_iv: "64bf968521b32e0ca4db49666ca41d08"
server_key_version: "encryptionkey@0"
I20260620 14:45:54.774500 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:45:54.775089 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:54.775241 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:45:54.775348 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "8f63de0b4612428290ea53de454a8967"
format_stamp: "Formatted at 2026-06-20 14:45:54 on dist-test-slave-cx89"
server_key: "926432ca057e2455cdffedf860a0fa8f"
server_key_iv: "64bf968521b32e0ca4db49666ca41d08"
server_key_version: "encryptionkey@0"
I20260620 14:45:54.775406 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:45:54.784371 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:45:54.786091 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:45:54.786213 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:45:54.786407 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:45:54.786715 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:45:54.786755 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:54.786784 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:45:54.786808 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:54.792223 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37379
I20260620 14:45:54.792280 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37379 every 8 connection(s)
I20260620 14:45:54.792563 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:45:54.792735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 1024
I20260620 14:45:54.792824 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:45:54.792999 392 external_mini_cluster.cc:1428] Setting key b84e18e02f540e7fe7d5c7d24a8ad0a5
Jun 20 14:45:54 dist-test-slave-cx89 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:45:54.802565 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:54.793351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36179 (local address 127.0.98.62:34091)
0620 14:45:54.793420 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:54.793423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:54.793619 (+ 196us) server_negotiation.cc:407] Connection header received
0620 14:45:54.794292 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:54.794293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:54.794304 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:54.794339 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:54.795529 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:54.796241 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:54.797324 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:54.797634 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:54.797704 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:54.800099 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:54.800100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:54.800101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:54.800107 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:54.801429 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:54.801803 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:54.801804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:54.801804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:54.801856 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:54.802090 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:54.802090 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:54.802091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:54.802185 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:54.802228 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:54.802437 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 14:45:54.802499 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:45:54.802743 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:54.793341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34091 (local address 127.0.98.1:36179)
0620 14:45:54.793496 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:54.793540 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:54.794175 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:54.794429 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:54.794433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:54.795171 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:54.795377 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:54.795383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:54.796351 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:54.796352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:54.797232 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:54.797234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:54.797711 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:54.797712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:54.797808 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:54.798372 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:54.798380 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:54.800003 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:54.801506 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:54.801512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:54.801514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:54.801742 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:54.801929 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:54.801930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:54.801930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:54.802035 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:54.802227 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:54.802229 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:54.802347 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 14:45:54.802425 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:45:54.802548 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:45:54.803181 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34091
I20260620 14:45:54.803290 1153 heartbeater.cc:464] Registering TS with master...
I20260620 14:45:54.803503 1153 heartbeater.cc:511] Master 127.0.98.62:34091 requested a full tablet report, sending...
I20260620 14:45:54.804014 962 ts_manager.cc:194] Registered new tserver with Master: 8f63de0b4612428290ea53de454a8967 (127.0.98.1:37379)
I20260620 14:45:54.804600 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:45:54.817804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34091
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:45:54.895004 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:45:54.895211 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:45:54.895243 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:45:54.896416 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:45:54.896456 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:45:54.896493 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:45:54.897908 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34091
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:45:54.898139 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:45:54.898334 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:45:54.903141 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:54.903218 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:54.903175 1158 server_base.cc:1034] running on GCE node
W20260620 14:45:54.903354 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:54.903570 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:45:54.903813 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:45:54.904955 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781966754904944 us; error 24 us; skew 500 ppm
Jun 20 14:45:54 dist-test-slave-cx89 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:45:54.907289 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:45:54.907717 1158 webserver.cc:460] Webserver started at http://127.0.98.2:43893/ using document root <none> and password file <none>
I20260620 14:45:54.907927 1158 fs_manager.cc:359] Metadata directory not provided
I20260620 14:45:54.907974 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:45:54.908094 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:45:54.909101 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "baeb9fe4d3ba495699f0141b466b94d5"
format_stamp: "Formatted at 2026-06-20 14:45:54 on dist-test-slave-cx89"
server_key: "619cfbff8f84ddaadf477a6cf3534c23"
server_key_iv: "aea84c5f19b8c9064f4719376bb14601"
server_key_version: "encryptionkey@0"
I20260620 14:45:54.909423 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "baeb9fe4d3ba495699f0141b466b94d5"
format_stamp: "Formatted at 2026-06-20 14:45:54 on dist-test-slave-cx89"
server_key: "619cfbff8f84ddaadf477a6cf3534c23"
server_key_iv: "aea84c5f19b8c9064f4719376bb14601"
server_key_version: "encryptionkey@0"
I20260620 14:45:54.910969 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:45:54.911662 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:54.911803 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:45:54.911881 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "baeb9fe4d3ba495699f0141b466b94d5"
format_stamp: "Formatted at 2026-06-20 14:45:54 on dist-test-slave-cx89"
server_key: "619cfbff8f84ddaadf477a6cf3534c23"
server_key_iv: "aea84c5f19b8c9064f4719376bb14601"
server_key_version: "encryptionkey@0"
I20260620 14:45:54.911947 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:45:54.924422 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:45:54.926395 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:45:54.926520 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:45:54.926712 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:45:54.927000 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:45:54.927032 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:54.927059 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:45:54.927083 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:54.932516 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34637
I20260620 14:45:54.932569 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34637 every 8 connection(s)
I20260620 14:45:54.932853 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:45:54 dist-test-slave-cx89 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:45:54.941267 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:54.933494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42961 (local address 127.0.98.62:34091)
0620 14:45:54.933550 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:54.933552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:54.933738 (+ 186us) server_negotiation.cc:407] Connection header received
0620 14:45:54.934225 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:54.934226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:54.934236 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:54.934302 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:54.935153 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:54.935689 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:54.936633 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:54.936826 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:54.936890 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:54.938930 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:54.938931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:54.938931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:54.938938 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:54.940128 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:54.940477 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:54.940478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:54.940479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:54.940538 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:54.940760 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:54.940771 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:54.940771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:54.940891 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:54.940949 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:54.941144 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 14:45:54.941202 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:45:54.941408 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:54.933475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34091 (local address 127.0.98.2:42961)
0620 14:45:54.933621 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:54.933661 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:54.934136 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:54.934356 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:54.934360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:54.934883 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:54.935068 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:54.935073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:54.935773 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:54.935774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:54.936554 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:54.936556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:54.936927 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:54.936929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:54.937054 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:54.937537 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:54.937544 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:54.938822 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:54.940202 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:54.940207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:54.940209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:54.940403 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:54.940599 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:54.940599 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:54.940600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:54.940691 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:54.940943 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:54.940946 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:54.941053 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 14:45:54.941125 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:45:54.941236 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260620 14:45:54.941824 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34091
I20260620 14:45:54.941915 1287 heartbeater.cc:464] Registering TS with master...
I20260620 14:45:54.942075 1287 heartbeater.cc:511] Master 127.0.98.62:34091 requested a full tablet report, sending...
I20260620 14:45:54.942111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 1158
I20260620 14:45:54.942193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:45:54.942350 392 external_mini_cluster.cc:1428] Setting key 4bb6d1d5a5aef780f56d5046d9796609
I20260620 14:45:54.942430 962 ts_manager.cc:194] Registered new tserver with Master: baeb9fe4d3ba495699f0141b466b94d5 (127.0.98.2:34637)
I20260620 14:45:54.942912 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42961
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:45:54.964447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34091
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:45:55.044107 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:45:55.044345 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:45:55.044371 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:45:55.045504 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:45:55.045542 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:45:55.045569 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:45:55.046948 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34091
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:45:55.047144 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:45:55.047340 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:45:55.052224 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:55.052282 1292 server_base.cc:1034] running on GCE node
W20260620 14:45:55.052198 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:55.052193 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:55.052608 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:45:55.052868 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:45:55.054014 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781966755053994 us; error 31 us; skew 500 ppm
Jun 20 14:45:55 dist-test-slave-cx89 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:45:55.056353 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:45:55.056727 1292 webserver.cc:460] Webserver started at http://127.0.98.3:34499/ using document root <none> and password file <none>
I20260620 14:45:55.056913 1292 fs_manager.cc:359] Metadata directory not provided
I20260620 14:45:55.056952 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:45:55.057040 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:45:55.058044 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "f62bbfbb54b9493ca2df3cc067bb2bbd"
format_stamp: "Formatted at 2026-06-20 14:45:55 on dist-test-slave-cx89"
server_key: "6d3aad8ed2e5fa3c84158bacfe431f5e"
server_key_iv: "416ef0a254a5752d856a2a54f361461a"
server_key_version: "encryptionkey@0"
I20260620 14:45:55.058362 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f62bbfbb54b9493ca2df3cc067bb2bbd"
format_stamp: "Formatted at 2026-06-20 14:45:55 on dist-test-slave-cx89"
server_key: "6d3aad8ed2e5fa3c84158bacfe431f5e"
server_key_iv: "416ef0a254a5752d856a2a54f361461a"
server_key_version: "encryptionkey@0"
I20260620 14:45:55.059587 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:45:55.060225 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:55.060400 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:45:55.060485 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "f62bbfbb54b9493ca2df3cc067bb2bbd"
format_stamp: "Formatted at 2026-06-20 14:45:55 on dist-test-slave-cx89"
server_key: "6d3aad8ed2e5fa3c84158bacfe431f5e"
server_key_iv: "416ef0a254a5752d856a2a54f361461a"
server_key_version: "encryptionkey@0"
I20260620 14:45:55.060551 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:45:55.071166 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:45:55.072875 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:45:55.073005 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:45:55.073199 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:45:55.073485 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:45:55.073523 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:55.073551 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:45:55.073575 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:55.079043 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37991
I20260620 14:45:55.079088 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37991 every 8 connection(s)
I20260620 14:45:55.079396 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:45:55 dist-test-slave-cx89 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:45:55.088316 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:55.080044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51073 (local address 127.0.98.62:34091)
0620 14:45:55.080092 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:55.080094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:55.080290 (+ 196us) server_negotiation.cc:407] Connection header received
0620 14:45:55.080748 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:55.080749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:55.080758 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:55.080805 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:55.081655 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.082136 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.083119 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.083309 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.083362 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:55.085655 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:55.085656 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:55.085656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:55.085664 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:55.087080 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:55.087452 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:55.087454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:55.087454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:55.087510 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:55.087726 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:55.087727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:55.087728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:55.087838 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:55.087891 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:55.088150 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 14:45:55.088226 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:45:55.088483 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:55.080030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34091 (local address 127.0.98.3:51073)
0620 14:45:55.080182 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:55.080215 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:55.080679 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:55.080863 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:55.080866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:55.081403 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:55.081591 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:55.081596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.082209 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.082210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:55.083042 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:55.083044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.083371 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.083371 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:55.083497 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:55.083902 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:55.083909 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:55.085567 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:55.087144 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:55.087151 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:55.087154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:55.087369 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:55.087565 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:55.087565 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:55.087566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:55.087671 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:55.087890 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:55.087905 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:55.088035 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 14:45:55.088124 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:45:55.088276 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260620 14:45:55.088678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 1292
I20260620 14:45:55.088768 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:45:55.088922 392 external_mini_cluster.cc:1428] Setting key 471087a4f8cfd016ae3fa186d4693574
I20260620 14:45:55.088919 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34091
I20260620 14:45:55.089006 1421 heartbeater.cc:464] Registering TS with master...
I20260620 14:45:55.089188 1421 heartbeater.cc:511] Master 127.0.98.62:34091 requested a full tablet report, sending...
I20260620 14:45:55.089519 962 ts_manager.cc:194] Registered new tserver with Master: f62bbfbb54b9493ca2df3cc067bb2bbd (127.0.98.3:37991)
I20260620 14:45:55.089843 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:45:55.089931 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51073
I20260620 14:45:55.096519 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:55.091333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42236 (local address 127.0.98.62:34091)
0620 14:45:55.091367 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:55.091369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:55.091494 (+ 125us) server_negotiation.cc:407] Connection header received
0620 14:45:55.091550 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:55.091551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:55.091561 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:55.091600 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:55.092094 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.092558 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.093234 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.093428 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.093477 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:55.094488 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:55.094489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:55.094490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:55.094496 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:55.095560 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:55.095840 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:55.095841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:55.095841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:55.095891 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:55.096045 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:55.096046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:55.096046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:55.096181 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:55.096264 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:55.096363 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 14:45:55.096436 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 14:45:55.098364 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:45:55.099941 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:45:55.105789 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:55.101972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.0.98.3:37991)
0620 14:45:55.102136 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:55.102141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:55.102152 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:45:55.102198 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:55.102200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:55.102216 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:55.102306 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:45:55.102486 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.103282 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.104742 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.105337 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.105377 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:55.105416 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:55.105670 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 14:45:55.105712 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
I20260620 14:45:55.105810 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:55.101834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34637 (local address 127.0.0.1:32944)
0620 14:45:55.102000 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:55.102040 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:55.102108 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:55.102425 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:55.102426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:55.102429 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:45:55.102485 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:55.102486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.103571 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.103572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:55.104695 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:55.104698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.105587 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.105588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:55.105646 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:55.105650 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:45:55.105678 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 14:45:55.105716 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":31,"thread_start_us":51,"threads_started":1}
I20260620 14:45:55.105808 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:55.101885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37991 (local address 127.0.0.1:34808)
0620 14:45:55.102089 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:55.102099 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:55.102145 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:55.102329 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:55.102332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:55.102334 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:45:55.102412 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:55.102413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.103365 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.103366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:55.104619 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:55.104622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.105468 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.105468 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:55.105528 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:55.105537 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 14:45:55.105630 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:45:55.105686 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":50,"threads_started":1}
I20260620 14:45:55.106300 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:55.101934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32944 (local address 127.0.98.2:34637)
0620 14:45:55.102255 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:55.102259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:55.102268 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:45:55.102305 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:55.102307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:55.102322 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:55.102422 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:45:55.102542 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.103450 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.104784 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.105492 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.105848 (+ 356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:55.106157 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:55.106184 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 14:45:55.106227 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":82,"threads_started":1}
I20260620 14:45:55.106786 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 022e25ff006b4320853e942373c497ba (DEFAULT_TABLE table=test-table [id=09fcbcd525e64040b014457c6d9f31ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:45:55.107118 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 022e25ff006b4320853e942373c497ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:55.107147 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 022e25ff006b4320853e942373c497ba (DEFAULT_TABLE table=test-table [id=09fcbcd525e64040b014457c6d9f31ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:45:55.107553 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 022e25ff006b4320853e942373c497ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:55.108047 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:55.102030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37379 (local address 127.0.0.1:50854)
0620 14:45:55.102353 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:55.102361 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:55.102402 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:55.103570 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:55.103571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:55.103573 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:45:55.103610 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:55.103611 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.104392 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.104393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:55.105518 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:55.105520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.107810 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.107811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:55.107870 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:55.107875 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:45:55.107942 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:45:55.107981 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":31,"threads_started":1}
I20260620 14:45:55.108233 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:55.102200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.0.98.1:37379)
0620 14:45:55.103439 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:55.103443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:55.103452 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:45:55.103487 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:55.103489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:55.103501 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:55.103565 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:45:55.103661 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.104312 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.105605 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.107628 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.107705 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:55.107772 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:55.107973 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 14:45:55.108020 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1183,"thread_start_us":82,"threads_started":1}
I20260620 14:45:55.108880 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 022e25ff006b4320853e942373c497ba (DEFAULT_TABLE table=test-table [id=09fcbcd525e64040b014457c6d9f31ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:45:55.109184 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 022e25ff006b4320853e942373c497ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:55.110144 1435 tablet_bootstrap.cc:492] T 022e25ff006b4320853e942373c497ba P baeb9fe4d3ba495699f0141b466b94d5: Bootstrap starting.
I20260620 14:45:55.110467 1436 tablet_bootstrap.cc:492] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd: Bootstrap starting.
I20260620 14:45:55.110704 1435 tablet_bootstrap.cc:654] T 022e25ff006b4320853e942373c497ba P baeb9fe4d3ba495699f0141b466b94d5: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:55.111105 1436 tablet_bootstrap.cc:654] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:55.111133 1435 log.cc:824] T 022e25ff006b4320853e942373c497ba P baeb9fe4d3ba495699f0141b466b94d5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:45:55.112062 1435 tablet_bootstrap.cc:492] T 022e25ff006b4320853e942373c497ba P baeb9fe4d3ba495699f0141b466b94d5: No bootstrap required, opened a new log
I20260620 14:45:55.112130 1435 ts_tablet_manager.cc:1397] T 022e25ff006b4320853e942373c497ba P baeb9fe4d3ba495699f0141b466b94d5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:45:55.112119 1436 log.cc:824] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd: Log is configured to *not* fsync() on all Append() calls
I20260620 14:45:55.112401 1438 tablet_bootstrap.cc:492] T 022e25ff006b4320853e942373c497ba P 8f63de0b4612428290ea53de454a8967: Bootstrap starting.
I20260620 14:45:55.112761 1436 tablet_bootstrap.cc:492] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd: No bootstrap required, opened a new log
I20260620 14:45:55.112826 1436 ts_tablet_manager.cc:1397] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:45:55.112972 1438 tablet_bootstrap.cc:654] T 022e25ff006b4320853e942373c497ba P 8f63de0b4612428290ea53de454a8967: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:55.113476 1438 log.cc:824] T 022e25ff006b4320853e942373c497ba P 8f63de0b4612428290ea53de454a8967: Log is configured to *not* fsync() on all Append() calls
I20260620 14:45:55.113684 1435 raft_consensus.cc:348] T 022e25ff006b4320853e942373c497ba P baeb9fe4d3ba495699f0141b466b94d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb9fe4d3ba495699f0141b466b94d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } } peers { permanent_uuid: "8f63de0b4612428290ea53de454a8967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37379 } } peers { permanent_uuid: "f62bbfbb54b9493ca2df3cc067bb2bbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37991 } }
I20260620 14:45:55.113889 1435 raft_consensus.cc:374] T 022e25ff006b4320853e942373c497ba P baeb9fe4d3ba495699f0141b466b94d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:55.113932 1435 raft_consensus.cc:729] T 022e25ff006b4320853e942373c497ba P baeb9fe4d3ba495699f0141b466b94d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baeb9fe4d3ba495699f0141b466b94d5, State: Initialized, Role: FOLLOWER
I20260620 14:45:55.114069 1435 consensus_queue.cc:260] T 022e25ff006b4320853e942373c497ba P baeb9fe4d3ba495699f0141b466b94d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb9fe4d3ba495699f0141b466b94d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } } peers { permanent_uuid: "8f63de0b4612428290ea53de454a8967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37379 } } peers { permanent_uuid: "f62bbfbb54b9493ca2df3cc067bb2bbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37991 } }
I20260620 14:45:55.114295 1438 tablet_bootstrap.cc:492] T 022e25ff006b4320853e942373c497ba P 8f63de0b4612428290ea53de454a8967: No bootstrap required, opened a new log
I20260620 14:45:55.114341 1435 ts_tablet_manager.cc:1428] T 022e25ff006b4320853e942373c497ba P baeb9fe4d3ba495699f0141b466b94d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:45:55.114414 1438 ts_tablet_manager.cc:1397] T 022e25ff006b4320853e942373c497ba P 8f63de0b4612428290ea53de454a8967: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:45:55.114462 1287 heartbeater.cc:503] Master 127.0.98.62:34091 was elected leader, sending a full tablet report...
I20260620 14:45:55.114630 1436 raft_consensus.cc:348] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb9fe4d3ba495699f0141b466b94d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } } peers { permanent_uuid: "8f63de0b4612428290ea53de454a8967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37379 } } peers { permanent_uuid: "f62bbfbb54b9493ca2df3cc067bb2bbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37991 } }
I20260620 14:45:55.114785 1436 raft_consensus.cc:374] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:55.114815 1436 raft_consensus.cc:729] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f62bbfbb54b9493ca2df3cc067bb2bbd, State: Initialized, Role: FOLLOWER
I20260620 14:45:55.114914 1436 consensus_queue.cc:260] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb9fe4d3ba495699f0141b466b94d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } } peers { permanent_uuid: "8f63de0b4612428290ea53de454a8967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37379 } } peers { permanent_uuid: "f62bbfbb54b9493ca2df3cc067bb2bbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37991 } }
I20260620 14:45:55.115144 1436 ts_tablet_manager.cc:1428] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 14:45:55.115271 1421 heartbeater.cc:503] Master 127.0.98.62:34091 was elected leader, sending a full tablet report...
I20260620 14:45:55.116258 1438 raft_consensus.cc:348] T 022e25ff006b4320853e942373c497ba P 8f63de0b4612428290ea53de454a8967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb9fe4d3ba495699f0141b466b94d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } } peers { permanent_uuid: "8f63de0b4612428290ea53de454a8967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37379 } } peers { permanent_uuid: "f62bbfbb54b9493ca2df3cc067bb2bbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37991 } }
I20260620 14:45:55.116375 1438 raft_consensus.cc:374] T 022e25ff006b4320853e942373c497ba P 8f63de0b4612428290ea53de454a8967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:55.116394 1438 raft_consensus.cc:729] T 022e25ff006b4320853e942373c497ba P 8f63de0b4612428290ea53de454a8967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f63de0b4612428290ea53de454a8967, State: Initialized, Role: FOLLOWER
I20260620 14:45:55.116495 1438 consensus_queue.cc:260] T 022e25ff006b4320853e942373c497ba P 8f63de0b4612428290ea53de454a8967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb9fe4d3ba495699f0141b466b94d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } } peers { permanent_uuid: "8f63de0b4612428290ea53de454a8967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37379 } } peers { permanent_uuid: "f62bbfbb54b9493ca2df3cc067bb2bbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37991 } }
I20260620 14:45:55.116710 1438 ts_tablet_manager.cc:1428] T 022e25ff006b4320853e942373c497ba P 8f63de0b4612428290ea53de454a8967: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:45:55.116734 1153 heartbeater.cc:503] Master 127.0.98.62:34091 was elected leader, sending a full tablet report...
W20260620 14:45:55.183388 1288 tablet.cc:2369] T 022e25ff006b4320853e942373c497ba P baeb9fe4d3ba495699f0141b466b94d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:45:55.293259 1154 tablet.cc:2369] T 022e25ff006b4320853e942373c497ba P 8f63de0b4612428290ea53de454a8967: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:45:55.330099 1422 tablet.cc:2369] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:45:55.363674 1442 raft_consensus.cc:482] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:45:55.363816 1442 raft_consensus.cc:504] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb9fe4d3ba495699f0141b466b94d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } } peers { permanent_uuid: "8f63de0b4612428290ea53de454a8967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37379 } } peers { permanent_uuid: "f62bbfbb54b9493ca2df3cc067bb2bbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37991 } }
I20260620 14:45:55.364207 1442 leader_election.cc:288] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baeb9fe4d3ba495699f0141b466b94d5 (127.0.98.2:34637), 8f63de0b4612428290ea53de454a8967 (127.0.98.1:37379)
I20260620 14:45:55.367565 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:55.364479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59819 (local address 127.0.98.2:34637)
0620 14:45:55.364522 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:55.364524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:55.364539 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:45:55.364572 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:55.364574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:55.364584 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:55.364653 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:45:55.364871 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.365545 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.366518 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.367195 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.367279 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:55.367290 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:55.367454 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 14:45:55.367496 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:45:55.367566 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:55.364409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34637 (local address 127.0.98.3:59819)
0620 14:45:55.364458 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:55.364470 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:55.364514 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:55.364707 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:55.364708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:55.364710 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:45:55.364779 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:55.364781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.365611 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.365612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:55.366429 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:55.366431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.367279 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.367280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:55.367361 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:55.367365 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:45:55.367447 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:45:55.367488 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 14:45:55.368047 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:55.364654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37875 (local address 127.0.98.1:37379)
0620 14:45:55.364715 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:55.364718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:55.365240 (+ 522us) server_negotiation.cc:407] Connection header received
0620 14:45:55.365307 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:55.365308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:55.365320 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:55.365371 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:45:55.365505 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.366214 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.367147 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.367675 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.367741 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:55.367752 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:55.367948 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 14:45:55.367991 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:45:55.368058 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:55.364511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37379 (local address 127.0.98.3:37875)
0620 14:45:55.364658 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:55.364666 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:55.365241 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:55.365380 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:55.365381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:55.365383 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:45:55.365450 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:55.365452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.366305 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.366306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:55.367067 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:55.367068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.367765 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.367766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:55.367850 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:55.367854 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:45:55.367934 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:45:55.367976 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260620 14:45:55.368046 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022e25ff006b4320853e942373c497ba" candidate_uuid: "f62bbfbb54b9493ca2df3cc067bb2bbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baeb9fe4d3ba495699f0141b466b94d5" is_pre_election: true
I20260620 14:45:55.368202 1241 raft_consensus.cc:2454] T 022e25ff006b4320853e942373c497ba P baeb9fe4d3ba495699f0141b466b94d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f62bbfbb54b9493ca2df3cc067bb2bbd in term 0.
I20260620 14:45:55.368410 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022e25ff006b4320853e942373c497ba" candidate_uuid: "f62bbfbb54b9493ca2df3cc067bb2bbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f63de0b4612428290ea53de454a8967" is_pre_election: true
I20260620 14:45:55.368547 1107 raft_consensus.cc:2454] T 022e25ff006b4320853e942373c497ba P 8f63de0b4612428290ea53de454a8967 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f62bbfbb54b9493ca2df3cc067bb2bbd in term 0.
I20260620 14:45:55.368752 1309 leader_election.cc:302] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: baeb9fe4d3ba495699f0141b466b94d5, f62bbfbb54b9493ca2df3cc067bb2bbd; no voters:
I20260620 14:45:55.368942 1442 raft_consensus.cc:2781] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:45:55.368990 1442 raft_consensus.cc:482] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:45:55.369017 1442 raft_consensus.cc:3037] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:55.369920 1442 raft_consensus.cc:504] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb9fe4d3ba495699f0141b466b94d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } } peers { permanent_uuid: "8f63de0b4612428290ea53de454a8967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37379 } } peers { permanent_uuid: "f62bbfbb54b9493ca2df3cc067bb2bbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37991 } }
I20260620 14:45:55.370059 1442 leader_election.cc:288] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd [CANDIDATE]: Term 1 election: Requested vote from peers baeb9fe4d3ba495699f0141b466b94d5 (127.0.98.2:34637), 8f63de0b4612428290ea53de454a8967 (127.0.98.1:37379)
I20260620 14:45:55.370208 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022e25ff006b4320853e942373c497ba" candidate_uuid: "f62bbfbb54b9493ca2df3cc067bb2bbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baeb9fe4d3ba495699f0141b466b94d5"
I20260620 14:45:55.370260 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022e25ff006b4320853e942373c497ba" candidate_uuid: "f62bbfbb54b9493ca2df3cc067bb2bbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f63de0b4612428290ea53de454a8967"
I20260620 14:45:55.370298 1241 raft_consensus.cc:3037] T 022e25ff006b4320853e942373c497ba P baeb9fe4d3ba495699f0141b466b94d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:55.370323 1107 raft_consensus.cc:3037] T 022e25ff006b4320853e942373c497ba P 8f63de0b4612428290ea53de454a8967 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:55.371160 1107 raft_consensus.cc:2454] T 022e25ff006b4320853e942373c497ba P 8f63de0b4612428290ea53de454a8967 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f62bbfbb54b9493ca2df3cc067bb2bbd in term 1.
I20260620 14:45:55.371160 1241 raft_consensus.cc:2454] T 022e25ff006b4320853e942373c497ba P baeb9fe4d3ba495699f0141b466b94d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f62bbfbb54b9493ca2df3cc067bb2bbd in term 1.
I20260620 14:45:55.371306 1309 leader_election.cc:302] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: baeb9fe4d3ba495699f0141b466b94d5, f62bbfbb54b9493ca2df3cc067bb2bbd; no voters:
I20260620 14:45:55.371436 1442 raft_consensus.cc:2781] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:45:55.371589 1442 raft_consensus.cc:686] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd [term 1 LEADER]: Becoming Leader. State: Replica: f62bbfbb54b9493ca2df3cc067bb2bbd, State: Running, Role: LEADER
I20260620 14:45:55.371764 1442 consensus_queue.cc:237] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb9fe4d3ba495699f0141b466b94d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } } peers { permanent_uuid: "8f63de0b4612428290ea53de454a8967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37379 } } peers { permanent_uuid: "f62bbfbb54b9493ca2df3cc067bb2bbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37991 } }
I20260620 14:45:55.372630 961 catalog_manager.cc:5495] T 022e25ff006b4320853e942373c497ba P f62bbfbb54b9493ca2df3cc067bb2bbd reported cstate change: term changed from 0 to 1, leader changed from <none> to f62bbfbb54b9493ca2df3cc067bb2bbd (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f62bbfbb54b9493ca2df3cc067bb2bbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baeb9fe4d3ba495699f0141b466b94d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f63de0b4612428290ea53de454a8967" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f62bbfbb54b9493ca2df3cc067bb2bbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37991 } health_report { overall_health: HEALTHY } } }
Jun 20 14:45:55 dist-test-slave-cx89 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:45:55.393980 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:55.385726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.0.98.1:37379)
0620 14:45:55.385783 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:55.385785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:55.385904 (+ 119us) server_negotiation.cc:407] Connection header received
0620 14:45:55.385963 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:55.385964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:55.385974 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:55.386031 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:55.386598 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.387390 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.388097 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.388397 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.388466 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:55.390943 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:55.390952 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:55.390953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:55.390958 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:55.393034 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:55.393365 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:55.393368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:55.393368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:55.393438 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:55.393574 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:55.393575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:55.393575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:55.393703 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:55.393798 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:55.393846 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:45:55.393913 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Jun 20 14:45:55 dist-test-slave-cx89 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966755, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:45:55 dist-test-slave-cx89 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966755, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:45:55.414179 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:55.407996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50876 (local address 127.0.98.1:37379)
0620 14:45:55.408057 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:55.408059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:55.408121 (+ 62us) server_negotiation.cc:407] Connection header received
0620 14:45:55.408192 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:55.408193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:55.408203 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:55.408244 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:55.408753 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.409223 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.409898 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.410087 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.410138 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:55.412217 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:55.412219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:55.412219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:55.412225 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:55.413257 (+ 1032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:55.413542 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:55.413543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:55.413543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:55.413583 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:55.413723 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:55.413723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:55.413724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:55.413804 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:55.413865 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:55.414045 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 14:45:55.414104 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 14:45:55 dist-test-slave-cx89 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:45:55 dist-test-slave-cx89 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:45:55.435043 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:55.428375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50886 (local address 127.0.98.1:37379)
0620 14:45:55.428437 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:55.428439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:55.428527 (+ 88us) server_negotiation.cc:407] Connection header received
0620 14:45:55.428611 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:55.428612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:55.428624 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:55.428677 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:55.429239 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.429728 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.430451 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:55.430634 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:55.430689 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:55.432916 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:55.432917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:55.432917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:55.432924 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:55.434164 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:55.434479 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:55.434480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:55.434480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:55.434535 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:55.434675 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:55.434675 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:55.434676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:55.434764 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:55.434843 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:55.434919 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 14:45:55.434978 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 14:45:55.435487 1067 tablet_service.cc:2908] Rejecting scan request for tablet 022e25ff006b4320853e942373c497ba: Uninitialized: clean time has not yet been initialized
I20260620 14:45:55.436270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 1024
I20260620 14:45:55.441087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 1158
I20260620 14:45:55.445051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 1292
I20260620 14:45:55.448608 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 930
2026-06-20T14:45:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2050 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:45:55 dist-test-slave-cx89 krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:45:55 dist-test-slave-cx89 krb5kdc[1459](info): set up 2 sockets
Jun 20 14:45:55 dist-test-slave-cx89 krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260620 14:45:56.403167 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:45:56 dist-test-slave-cx89 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:45:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:45:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:45:56.513742 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44051
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:45:56.590847 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:45:56.591053 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:45:56.591081 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:45:56.592244 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:45:56.592283 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:45:56.592300 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:45:56.592316 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:45:56.592332 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:45:56.592348 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:45:56.593807 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44051
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:45:56.594063 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:45:56.594265 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:45:56.598742 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:56.598716 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:56.598783 1472 server_base.cc:1034] running on GCE node
W20260620 14:45:56.598742 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:56.599144 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:45:56.599386 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:45:56.600556 1472 hybrid_clock.cc:648] HybridClock initialized: now 1781966756600542 us; error 28 us; skew 500 ppm
Jun 20 14:45:56 dist-test-slave-cx89 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:45:56.602854 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:45:56.603230 1472 webserver.cc:460] Webserver started at http://127.0.98.62:42487/ using document root <none> and password file <none>
I20260620 14:45:56.603454 1472 fs_manager.cc:359] Metadata directory not provided
I20260620 14:45:56.603502 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:45:56.603617 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:45:56.604615 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "e681cf6156da467d933f61e14969fb9b"
format_stamp: "Formatted at 2026-06-20 14:45:56 on dist-test-slave-cx89"
server_key: "341239596279b3a0106ebb8c529d9b6b"
server_key_iv: "2aa48483e3f06a68d06df89b80dded07"
server_key_version: "encryptionkey@0"
I20260620 14:45:56.604939 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "e681cf6156da467d933f61e14969fb9b"
format_stamp: "Formatted at 2026-06-20 14:45:56 on dist-test-slave-cx89"
server_key: "341239596279b3a0106ebb8c529d9b6b"
server_key_iv: "2aa48483e3f06a68d06df89b80dded07"
server_key_version: "encryptionkey@0"
I20260620 14:45:56.606118 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:45:56.606689 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:56.606833 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:45:56.606911 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "e681cf6156da467d933f61e14969fb9b"
format_stamp: "Formatted at 2026-06-20 14:45:56 on dist-test-slave-cx89"
server_key: "341239596279b3a0106ebb8c529d9b6b"
server_key_iv: "2aa48483e3f06a68d06df89b80dded07"
server_key_version: "encryptionkey@0"
I20260620 14:45:56.606961 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:45:56.616720 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:45:56.618436 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:45:56.618542 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:45:56.622594 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44051
I20260620 14:45:56.622665 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44051 every 8 connection(s)
I20260620 14:45:56.622928 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:45:56.623628 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:56.625758 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b: Bootstrap starting.
I20260620 14:45:56.626318 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:56.626569 1550 log.cc:824] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:45:56.627173 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b: No bootstrap required, opened a new log
I20260620 14:45:56.627839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 1472
I20260620 14:45:56.627914 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:45:56.628062 392 external_mini_cluster.cc:1428] Setting key 1e3813734853998a3a4491a678b7b141
I20260620 14:45:56.629390 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e681cf6156da467d933f61e14969fb9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44051 } }
I20260620 14:45:56.629534 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:56.629617 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e681cf6156da467d933f61e14969fb9b, State: Initialized, Role: FOLLOWER
I20260620 14:45:56.629755 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e681cf6156da467d933f61e14969fb9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44051 } }
I20260620 14:45:56.629834 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:45:56.629869 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:45:56.629915 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:56.630513 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e681cf6156da467d933f61e14969fb9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44051 } }
I20260620 14:45:56.630601 1550 leader_election.cc:302] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e681cf6156da467d933f61e14969fb9b; no voters:
I20260620 14:45:56.630746 1550 leader_election.cc:288] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:45:56.630786 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:45:56.630934 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b [term 1 LEADER]: Becoming Leader. State: Replica: e681cf6156da467d933f61e14969fb9b, State: Running, Role: LEADER
I20260620 14:45:56.631006 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:45:56.631100 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e681cf6156da467d933f61e14969fb9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44051 } }
I20260620 14:45:56.631498 1555 sys_catalog.cc:455] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e681cf6156da467d933f61e14969fb9b. Latest consensus state: current_term: 1 leader_uuid: "e681cf6156da467d933f61e14969fb9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e681cf6156da467d933f61e14969fb9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44051 } } }
I20260620 14:45:56.631728 1555 sys_catalog.cc:458] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b [sys.catalog]: This master's current role is: LEADER
I20260620 14:45:56.631664 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e681cf6156da467d933f61e14969fb9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e681cf6156da467d933f61e14969fb9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44051 } } }
I20260620 14:45:56.632123 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b [sys.catalog]: This master's current role is: LEADER
I20260620 14:45:56.632284 1564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:45:56.632897 1564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 14:45:56 dist-test-slave-cx89 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:45:56.635905 1564 catalog_manager.cc:1347] Generated new cluster ID: b79dc0c12edc46f886e37254168441fc
I20260620 14:45:56.635953 1564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:45:56.639108 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:56.628527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54546 (local address 127.0.98.62:44051)
0620 14:45:56.628668 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:56.628676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:56.628701 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:45:56.629364 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:56.629370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:56.629418 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:56.629540 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:56.630259 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:56.631521 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:56.632577 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:56.633004 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:56.633092 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:56.635677 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:56.635681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:56.635681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:56.635693 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:56.637554 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:56.637961 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:56.637964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:56.637964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:56.638077 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:56.638265 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:56.638266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:56.638266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:56.638549 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:56.638682 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:56.638788 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 14:45:56.638913 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":53,"threads_started":1}
I20260620 14:45:56.649916 1564 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:45:56.650414 1564 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:45:56.660606 1564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b: Generated new TSK 0
I20260620 14:45:56.660804 1564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:45:56.690027 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44051
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:45:56.769747 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:45:56.769942 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:45:56.769966 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:45:56.771059 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:45:56.771092 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:45:56.771111 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:45:56.771126 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:45:56.772610 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:45:56.772787 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:45:56.772960 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:45:56.777559 1576 server_base.cc:1034] running on GCE node
W20260620 14:45:56.777601 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:56.777519 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:56.777771 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:56.777998 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:45:56.778185 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:45:56.779307 1576 hybrid_clock.cc:648] HybridClock initialized: now 1781966756779300 us; error 26 us; skew 500 ppm
Jun 20 14:45:56 dist-test-slave-cx89 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:45:56.781533 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:45:56.781908 1576 webserver.cc:460] Webserver started at http://127.0.98.1:44489/ using document root <none> and password file <none>
I20260620 14:45:56.782071 1576 fs_manager.cc:359] Metadata directory not provided
I20260620 14:45:56.782107 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:45:56.782194 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:45:56.783144 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "033c799c2a984f309f921ed0344e79c3"
format_stamp: "Formatted at 2026-06-20 14:45:56 on dist-test-slave-cx89"
server_key: "bccc62791d8b8f34655fc3b85ba32d51"
server_key_iv: "f606fb81746cbdf0a9be2d58ad7ed07c"
server_key_version: "encryptionkey@0"
I20260620 14:45:56.783468 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "033c799c2a984f309f921ed0344e79c3"
format_stamp: "Formatted at 2026-06-20 14:45:56 on dist-test-slave-cx89"
server_key: "bccc62791d8b8f34655fc3b85ba32d51"
server_key_iv: "f606fb81746cbdf0a9be2d58ad7ed07c"
server_key_version: "encryptionkey@0"
I20260620 14:45:56.784632 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:45:56.785266 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:56.785404 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:45:56.785470 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "033c799c2a984f309f921ed0344e79c3"
format_stamp: "Formatted at 2026-06-20 14:45:56 on dist-test-slave-cx89"
server_key: "bccc62791d8b8f34655fc3b85ba32d51"
server_key_iv: "f606fb81746cbdf0a9be2d58ad7ed07c"
server_key_version: "encryptionkey@0"
I20260620 14:45:56.785516 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:45:56.801995 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:45:56.803689 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:45:56.803809 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:45:56.804268 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:45:56.804314 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:56.804344 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:45:56.804368 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:56.810374 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32885
I20260620 14:45:56.810484 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32885 every 8 connection(s)
I20260620 14:45:56.810758 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:45:56 dist-test-slave-cx89 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:45:56.814083 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:56.804699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36809 (local address 127.0.98.62:44051)
0620 14:45:56.804769 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:56.804771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:56.804908 (+ 137us) server_negotiation.cc:407] Connection header received
0620 14:45:56.805363 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:56.805364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:56.805377 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:56.805449 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:56.806187 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:56.806947 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:56.808418 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:56.808712 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:56.808769 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:56.811778 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:56.811780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:56.811780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:56.811787 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:56.812857 (+ 1070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:56.813194 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:56.813195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:56.813196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:56.813249 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:56.813457 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:56.813458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:56.813458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:56.813567 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:56.813619 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:56.813861 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 14:45:56.813954 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:45:56.814200 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:56.804656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44051 (local address 127.0.98.1:36809)
0620 14:45:56.804787 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:56.804826 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:56.805278 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:56.805522 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:56.805527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:56.805824 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:56.806089 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:56.806095 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:56.807034 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:56.807035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:56.808311 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:56.808314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:56.809535 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:56.809536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:56.809687 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:56.810194 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:56.810204 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:56.811691 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:56.812926 (+ 1235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:56.812933 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:56.812935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:56.813134 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:56.813300 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:56.813300 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:56.813301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:56.813406 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:56.813617 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:56.813623 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:56.813751 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 14:45:56.813834 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:45:56.813976 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":70,"threads_started":1}
I20260620 14:45:56.814234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 1576
I20260620 14:45:56.814328 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:45:56.814482 392 external_mini_cluster.cc:1428] Setting key 96e6485337a1a51e4f75e9927189077b
I20260620 14:45:56.814702 1707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44051
I20260620 14:45:56.814783 1707 heartbeater.cc:464] Registering TS with master...
I20260620 14:45:56.814955 1707 heartbeater.cc:511] Master 127.0.98.62:44051 requested a full tablet report, sending...
I20260620 14:45:56.815477 1503 ts_manager.cc:194] Registered new tserver with Master: 033c799c2a984f309f921ed0344e79c3 (127.0.98.1:32885)
I20260620 14:45:56.816272 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36809
I20260620 14:45:56.820916 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:56.815550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44051 (local address 127.0.98.1:55023)
0620 14:45:56.815723 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:56.815731 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:56.815782 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:56.815903 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:56.815904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:56.816060 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:56.816102 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:56.816103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:56.816891 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:56.816891 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:56.817394 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:56.817395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:56.817791 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:56.817792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:56.817834 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:56.818223 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:56.818227 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:56.818479 (+ 252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:56.819996 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:56.819997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:56.819997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:56.820158 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:56.820380 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:56.820380 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:56.820381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:56.820433 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:56.820670 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:56.820671 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:56.820707 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:45:56.820761 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:45:56.820837 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":63,"threads_started":1}
I20260620 14:45:56.820950 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:56.815661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55023 (local address 127.0.98.62:44051)
0620 14:45:56.815696 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:56.815697 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:56.815770 (+ 73us) server_negotiation.cc:407] Connection header received
0620 14:45:56.815832 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:56.815833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:56.815842 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:56.815891 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:56.816162 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:56.816790 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:56.817464 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:56.817731 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:56.817797 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:56.818551 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:56.818552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:56.818553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:56.818559 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:56.819903 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:56.820226 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:56.820227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:56.820227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:56.820284 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:56.820489 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:56.820490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:56.820491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:56.820612 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:56.820682 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:56.820784 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 14:45:56.820855 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:45:56.839890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44051
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:45:56.917130 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:45:56.917318 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:45:56.917341 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:45:56.918430 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:45:56.918462 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:45:56.918480 1716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:45:56.918496 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:45:56.919945 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:45:56.920120 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:45:56.920280 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:45:56.924782 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:56.924866 1716 server_base.cc:1034] running on GCE node
W20260620 14:45:56.924754 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:56.924940 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:56.925127 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:45:56.925347 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:45:56.926483 1716 hybrid_clock.cc:648] HybridClock initialized: now 1781966756926475 us; error 34 us; skew 500 ppm
Jun 20 14:45:56 dist-test-slave-cx89 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:45:56.928740 1716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:45:56.929140 1716 webserver.cc:460] Webserver started at http://127.0.98.2:45309/ using document root <none> and password file <none>
I20260620 14:45:56.929330 1716 fs_manager.cc:359] Metadata directory not provided
I20260620 14:45:56.929375 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:45:56.929492 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:45:56.930480 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4"
format_stamp: "Formatted at 2026-06-20 14:45:56 on dist-test-slave-cx89"
server_key: "57d3f21764295aa55197897807eb7928"
server_key_iv: "3c8c15cba1db5770ef0d82f886daae3c"
server_key_version: "encryptionkey@0"
I20260620 14:45:56.930831 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4"
format_stamp: "Formatted at 2026-06-20 14:45:56 on dist-test-slave-cx89"
server_key: "57d3f21764295aa55197897807eb7928"
server_key_iv: "3c8c15cba1db5770ef0d82f886daae3c"
server_key_version: "encryptionkey@0"
I20260620 14:45:56.932250 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:45:56.932927 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:56.933068 1716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:45:56.933146 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4"
format_stamp: "Formatted at 2026-06-20 14:45:56 on dist-test-slave-cx89"
server_key: "57d3f21764295aa55197897807eb7928"
server_key_iv: "3c8c15cba1db5770ef0d82f886daae3c"
server_key_version: "encryptionkey@0"
I20260620 14:45:56.933208 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:45:56.959628 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:45:56.961575 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:45:56.961715 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:45:56.962205 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:45:56.962242 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:56.962276 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:45:56.962301 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:56.968744 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42915
Jun 20 14:45:56 dist-test-slave-cx89 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:45:56.969092 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:45:56.969584 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42915 every 8 connection(s)
I20260620 14:45:56.971822 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:56.962715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41937 (local address 127.0.98.62:44051)
0620 14:45:56.962765 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:56.962767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:56.962994 (+ 227us) server_negotiation.cc:407] Connection header received
0620 14:45:56.963553 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:56.963555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:56.963569 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:56.963632 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:56.964424 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:56.965146 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:56.966638 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:56.966913 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:56.966982 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:56.969528 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:56.969529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:56.969530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:56.969536 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:56.970661 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:56.971039 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:56.971041 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:56.971041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:56.971099 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:56.971290 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:56.971291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:56.971291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:56.971391 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:56.971489 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:56.971695 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 14:45:56.971759 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:45:56.971948 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:56.962707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44051 (local address 127.0.98.2:41937)
0620 14:45:56.962872 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:56.962907 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:56.963454 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:56.963704 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:56.963709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:56.964064 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:56.964335 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:56.964343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:56.965244 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:56.965245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:56.966542 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:56.966545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:56.966996 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:56.966997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:56.967138 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:56.967714 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:56.967724 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:56.969442 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:56.970738 (+ 1296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:56.970744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:56.970745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:56.970954 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:56.971155 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:56.971156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:56.971156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:56.971245 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:56.971479 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:56.971483 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:56.971597 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 14:45:56.971675 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:45:56.971784 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260620 14:45:56.972376 1847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44051
I20260620 14:45:56.972460 1847 heartbeater.cc:464] Registering TS with master...
I20260620 14:45:56.972631 1847 heartbeater.cc:511] Master 127.0.98.62:44051 requested a full tablet report, sending...
I20260620 14:45:56.972982 1504 ts_manager.cc:194] Registered new tserver with Master: 9587a2e3bdbd42b48e3ab68e8b8916a4 (127.0.98.2:42915)
I20260620 14:45:56.973429 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41937
I20260620 14:45:56.974284 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 1716
I20260620 14:45:56.974368 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:45:56.974524 392 external_mini_cluster.cc:1428] Setting key 7df9d83d4e03708f7bbda3522dc15302
I20260620 14:45:56.978358 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:56.973292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59789 (local address 127.0.98.62:44051)
0620 14:45:56.973331 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:56.973333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:56.973441 (+ 108us) server_negotiation.cc:407] Connection header received
0620 14:45:56.973468 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:56.973469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:56.973478 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:56.973534 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:56.973897 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:56.974362 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:56.975149 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:56.975337 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:56.975385 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:56.976214 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:56.976215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:56.976215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:56.976221 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:56.977396 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:56.977723 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:56.977723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:56.977724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:56.977764 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:56.977917 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:56.977917 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:56.977917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:56.978046 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:56.978095 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:56.978235 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 14:45:56.978285 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:45:56.978411 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:56.973231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44051 (local address 127.0.98.2:59789)
0620 14:45:56.973380 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:56.973390 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:56.973438 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:56.973575 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:56.973577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:56.973771 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:56.973829 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:56.973831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:56.974443 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:56.974444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:56.975084 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:56.975086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:56.975387 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:56.975388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:56.975475 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:56.975829 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:56.975833 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:56.976137 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:56.977470 (+ 1333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:56.977471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:56.977472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:56.977662 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:56.977802 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:56.977802 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:56.977803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:56.977863 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:56.978103 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:56.978104 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:56.978147 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:45:56.978218 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:45:56.978313 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:45:56.996295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44051
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:45:57.073135 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:45:57.073348 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:45:57.073372 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:45:57.074491 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:45:57.074527 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:45:57.074544 1856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:45:57.074560 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:45:57.076009 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:45:57.076189 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:45:57.076354 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:45:57.080910 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:57.080938 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:57.080958 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:57.081135 1856 server_base.cc:1034] running on GCE node
I20260620 14:45:57.081322 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:45:57.081535 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:45:57.082655 1856 hybrid_clock.cc:648] HybridClock initialized: now 1781966757082645 us; error 38 us; skew 500 ppm
Jun 20 14:45:57 dist-test-slave-cx89 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:45:57.084892 1856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:45:57.085280 1856 webserver.cc:460] Webserver started at http://127.0.98.3:37561/ using document root <none> and password file <none>
I20260620 14:45:57.085451 1856 fs_manager.cc:359] Metadata directory not provided
I20260620 14:45:57.085492 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:45:57.085578 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:45:57.086517 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "c6e28b94711f499f993f8a61c6918bd8"
format_stamp: "Formatted at 2026-06-20 14:45:57 on dist-test-slave-cx89"
server_key: "8776e319b01c4d32e9a6205294db3840"
server_key_iv: "57700dd3978017319b9198280392c039"
server_key_version: "encryptionkey@0"
I20260620 14:45:57.086859 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c6e28b94711f499f993f8a61c6918bd8"
format_stamp: "Formatted at 2026-06-20 14:45:57 on dist-test-slave-cx89"
server_key: "8776e319b01c4d32e9a6205294db3840"
server_key_iv: "57700dd3978017319b9198280392c039"
server_key_version: "encryptionkey@0"
I20260620 14:45:57.088057 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:45:57.088728 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:57.088856 1856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:45:57.088933 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "c6e28b94711f499f993f8a61c6918bd8"
format_stamp: "Formatted at 2026-06-20 14:45:57 on dist-test-slave-cx89"
server_key: "8776e319b01c4d32e9a6205294db3840"
server_key_iv: "57700dd3978017319b9198280392c039"
server_key_version: "encryptionkey@0"
I20260620 14:45:57.088996 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:45:57.099187 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:45:57.101117 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:45:57.101269 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:45:57.101775 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:45:57.101825 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:57.101857 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:45:57.101888 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:45:57 dist-test-slave-cx89 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:45:57.109915 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34675
I20260620 14:45:57.109982 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34675 every 8 connection(s)
I20260620 14:45:57.110359 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:45:57.111095 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.102120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43485 (local address 127.0.98.62:44051)
0620 14:45:57.102182 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:57.102184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:57.102420 (+ 236us) server_negotiation.cc:407] Connection header received
0620 14:45:57.103095 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:57.103096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:57.103107 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:57.103147 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:57.104020 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.104530 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.105489 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.105683 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.105736 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.108225 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:57.108226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:57.108226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:57.108234 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:57.109603 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:57.110098 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:57.110099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:57.110099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:57.110159 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:57.110446 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:57.110447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:57.110447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:57.110568 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:57.110627 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:57.110950 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 14:45:57.111031 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:45:57.111203 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.102114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44051 (local address 127.0.98.3:43485)
0620 14:45:57.102279 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:57.102324 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:57.102984 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:57.103226 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:57.103231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:57.103614 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:57.103911 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.103918 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.104614 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.104615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.105406 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.105408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.105747 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.105749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.105860 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.106350 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:57.106358 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:57.108137 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:57.109690 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:57.109695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:57.109698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:57.110007 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:57.110228 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:57.110229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:57.110230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:57.110355 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:57.110655 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:57.110659 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:57.110812 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 14:45:57.110916 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:45:57.111035 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260620 14:45:57.111663 1987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44051
I20260620 14:45:57.111744 1987 heartbeater.cc:464] Registering TS with master...
I20260620 14:45:57.111912 1987 heartbeater.cc:511] Master 127.0.98.62:44051 requested a full tablet report, sending...
I20260620 14:45:57.112274 1503 ts_manager.cc:194] Registered new tserver with Master: c6e28b94711f499f993f8a61c6918bd8 (127.0.98.3:34675)
I20260620 14:45:57.112649 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43485
I20260620 14:45:57.117206 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.112612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49199 (local address 127.0.98.62:44051)
0620 14:45:57.112655 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:57.112657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:57.112701 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:45:57.112773 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:57.112773 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:57.112783 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:57.112844 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:57.113123 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.113577 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.114247 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.114435 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.114485 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.115234 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:57.115235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:57.115235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:57.115242 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:57.116471 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:57.116733 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:57.116734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:57.116734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:57.116777 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:57.116894 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:57.116894 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:57.116894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:57.116975 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:57.117015 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:57.117105 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 14:45:57.117153 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:45:57.117214 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.112545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44051 (local address 127.0.98.3:49199)
0620 14:45:57.112649 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:57.112659 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:57.112711 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:57.112849 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:57.112850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:57.113001 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:57.113055 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.113057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.113659 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.113660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.114179 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.114181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.114489 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.114490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.114552 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.114917 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:57.114921 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:57.115161 (+ 240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:57.116542 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:57.116543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:57.116543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:57.116685 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:57.116817 (+ 132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:57.116817 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:57.116817 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:57.116859 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:57.117011 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:57.117012 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:57.117043 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:45:57.117083 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 14:45:57.117148 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":60,"threads_started":1}
I20260620 14:45:57.120458 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 1856
I20260620 14:45:57.120534 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:45:57.120682 392 external_mini_cluster.cc:1428] Setting key ad5cc9339a366718c38c0a78bef1126a
I20260620 14:45:57.121472 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:45:57 dist-test-slave-cx89 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:45:57 dist-test-slave-cx89 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:45:57.150213 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.142685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54554 (local address 127.0.98.62:44051)
0620 14:45:57.142742 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:57.142744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:57.142753 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:45:57.142781 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:57.142782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:57.142792 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:57.142869 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:57.143589 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.144345 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.145369 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.145661 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.145717 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.147996 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:57.147997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:57.147997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:57.148004 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:57.149250 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:57.149564 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:57.149565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:57.149565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:57.149608 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:57.149731 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:57.149732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:57.149732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:57.149843 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:57.149919 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:57.150075 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 14:45:57.150134 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:45:57.151744 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:45:57.153339 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:45:57.159435 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.155396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50876 (local address 127.0.98.1:32885)
0620 14:45:57.155713 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:57.155717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:57.155726 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:45:57.155764 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:57.155767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:57.155781 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:57.155854 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:45:57.155996 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.157258 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.158515 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.159005 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.159077 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.159156 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:57.159278 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 14:45:57.159316 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":113,"threads_started":1}
I20260620 14:45:57.159399 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.155232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32885 (local address 127.0.0.1:50876)
0620 14:45:57.155510 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:57.155552 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:57.155622 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:57.155855 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:57.155858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:57.155860 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:45:57.155922 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.155924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.157336 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.157337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.158423 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.158425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.159085 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.159086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.159173 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.159187 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 14:45:57.159264 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:45:57.159325 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":26,"thread_start_us":408,"threads_started":1}
I20260620 14:45:57.159494 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.155439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59214 (local address 127.0.98.3:34675)
0620 14:45:57.155725 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:57.155729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:57.155827 (+ 98us) server_negotiation.cc:407] Connection header received
0620 14:45:57.155958 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:57.155960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:57.155976 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:57.156095 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:45:57.156184 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.157252 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.158274 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.159000 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.159069 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.159147 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:57.159332 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 14:45:57.159370 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":88,"threads_started":1}
I20260620 14:45:57.159616 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.155483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44498 (local address 127.0.98.2:42915)
0620 14:45:57.155639 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:57.155644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:57.155757 (+ 113us) server_negotiation.cc:407] Connection header received
0620 14:45:57.155836 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:57.155839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:57.155855 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:57.155932 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:45:57.156201 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.156978 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.158481 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.159143 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.159198 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.159258 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:57.159484 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 14:45:57.159521 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260620 14:45:57.159688 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.155234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42915 (local address 127.0.0.1:44498)
0620 14:45:57.155685 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:57.155695 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:57.155758 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:57.156013 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:57.156014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:57.156016 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:45:57.156060 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.156062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.157060 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.157061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.158398 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.158400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.159286 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.159287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.159339 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.159342 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:45:57.159381 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 14:45:57.159442 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":39,"thread_start_us":68,"threads_started":1}
I20260620 14:45:57.159754 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.155233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34675 (local address 127.0.0.1:59214)
0620 14:45:57.155768 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:57.155777 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:57.155851 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:57.156073 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:57.156074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:57.156075 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:45:57.156104 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.156105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.157346 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.157346 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.158137 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.158140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.159184 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.159185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.159244 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.159248 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:45:57.159611 (+ 363us) client_negotiation.cc:240] Negotiation successful
0620 14:45:57.159649 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":5,"thread_start_us":61,"threads_started":1}
I20260620 14:45:57.160418 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 8d778b5b704a4eb680e8d8eb24c15dfd (DEFAULT_TABLE table=test-table [id=e317168f3eca476f9841a5747f24e7cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:45:57.160570 1640 tablet_service.cc:1467] Processing CreateTablet for tablet 8d778b5b704a4eb680e8d8eb24c15dfd (DEFAULT_TABLE table=test-table [id=e317168f3eca476f9841a5747f24e7cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:45:57.160570 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 8d778b5b704a4eb680e8d8eb24c15dfd (DEFAULT_TABLE table=test-table [id=e317168f3eca476f9841a5747f24e7cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:45:57.160730 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d778b5b704a4eb680e8d8eb24c15dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:57.160910 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d778b5b704a4eb680e8d8eb24c15dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:57.160910 1640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d778b5b704a4eb680e8d8eb24c15dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:57.163290 2007 tablet_bootstrap.cc:492] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8: Bootstrap starting.
I20260620 14:45:57.163949 2007 tablet_bootstrap.cc:654] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:57.164212 2007 log.cc:824] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:45:57.164873 2007 tablet_bootstrap.cc:492] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8: No bootstrap required, opened a new log
I20260620 14:45:57.164930 2007 ts_tablet_manager.cc:1397] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:45:57.166214 2007 raft_consensus.cc:348] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } } peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } }
I20260620 14:45:57.166378 2007 raft_consensus.cc:374] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:57.166415 2007 raft_consensus.cc:729] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6e28b94711f499f993f8a61c6918bd8, State: Initialized, Role: FOLLOWER
I20260620 14:45:57.166525 2007 consensus_queue.cc:260] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } } peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } }
I20260620 14:45:57.166792 2007 ts_tablet_manager.cc:1428] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:45:57.166822 1987 heartbeater.cc:503] Master 127.0.98.62:44051 was elected leader, sending a full tablet report...
I20260620 14:45:57.166967 2009 tablet_bootstrap.cc:492] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4: Bootstrap starting.
I20260620 14:45:57.167622 2009 tablet_bootstrap.cc:654] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:57.167881 2009 log.cc:824] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:45:57.168154 2010 tablet_bootstrap.cc:492] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3: Bootstrap starting.
I20260620 14:45:57.168497 2009 tablet_bootstrap.cc:492] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4: No bootstrap required, opened a new log
I20260620 14:45:57.168552 2009 ts_tablet_manager.cc:1397] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:45:57.168654 2010 tablet_bootstrap.cc:654] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:57.168893 2010 log.cc:824] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:45:57.169451 2010 tablet_bootstrap.cc:492] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3: No bootstrap required, opened a new log
I20260620 14:45:57.169507 2010 ts_tablet_manager.cc:1397] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:45:57.170128 2009 raft_consensus.cc:348] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } } peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } }
I20260620 14:45:57.170248 2009 raft_consensus.cc:374] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:57.170267 2009 raft_consensus.cc:729] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9587a2e3bdbd42b48e3ab68e8b8916a4, State: Initialized, Role: FOLLOWER
I20260620 14:45:57.170370 2009 consensus_queue.cc:260] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } } peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } }
I20260620 14:45:57.170570 2009 ts_tablet_manager.cc:1428] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:45:57.170619 1847 heartbeater.cc:503] Master 127.0.98.62:44051 was elected leader, sending a full tablet report...
I20260620 14:45:57.171126 2010 raft_consensus.cc:348] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } } peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } }
I20260620 14:45:57.171274 2010 raft_consensus.cc:374] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:57.171317 2010 raft_consensus.cc:729] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 033c799c2a984f309f921ed0344e79c3, State: Initialized, Role: FOLLOWER
I20260620 14:45:57.171446 2010 consensus_queue.cc:260] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } } peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } }
I20260620 14:45:57.171700 2010 ts_tablet_manager.cc:1428] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:45:57.171744 1707 heartbeater.cc:503] Master 127.0.98.62:44051 was elected leader, sending a full tablet report...
I20260620 14:45:57.183729 2014 raft_consensus.cc:482] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:45:57.183856 2014 raft_consensus.cc:504] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } } peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } }
I20260620 14:45:57.184192 2014 leader_election.cc:288] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6e28b94711f499f993f8a61c6918bd8 (127.0.98.3:34675), 033c799c2a984f309f921ed0344e79c3 (127.0.98.1:32885)
I20260620 14:45:57.186805 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.184458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51097 (local address 127.0.98.1:32885)
0620 14:45:57.184497 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:57.184499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:57.184508 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:45:57.184535 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:57.184536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:57.184546 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:57.184590 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:45:57.184711 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.185173 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.186160 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.186521 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.186573 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.186582 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:57.186727 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 14:45:57.186753 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:45:57.186832 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.184372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32885 (local address 127.0.98.2:51097)
0620 14:45:57.184432 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:57.184443 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:57.184489 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:57.184592 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:57.184593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:57.184594 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:45:57.184658 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.184659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.185238 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.185239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.186045 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.186048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.186590 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.186591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.186666 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.186670 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:45:57.186724 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:45:57.186761 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:45:57.187177 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.184428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35029 (local address 127.0.98.3:34675)
0620 14:45:57.184473 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:57.184475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:57.184623 (+ 148us) server_negotiation.cc:407] Connection header received
0620 14:45:57.184672 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:57.184673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:57.184684 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:57.184731 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:45:57.184866 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.185312 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.186363 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.186786 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.186844 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.186852 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:57.187083 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 14:45:57.187119 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:45:57.187199 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.184381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34675 (local address 127.0.98.2:35029)
0620 14:45:57.184543 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:57.184552 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:57.184622 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:57.184742 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:57.184743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:57.184745 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:45:57.184815 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.184817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.185378 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.185379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.186285 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.186287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.186945 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.186946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.187018 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.187021 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:45:57.187075 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:45:57.187107 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":13,"thread_start_us":104,"threads_started":1}
I20260620 14:45:57.187178 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d778b5b704a4eb680e8d8eb24c15dfd" candidate_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "033c799c2a984f309f921ed0344e79c3" is_pre_election: true
I20260620 14:45:57.187328 1661 raft_consensus.cc:2454] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9587a2e3bdbd42b48e3ab68e8b8916a4 in term 0.
I20260620 14:45:57.187511 1940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d778b5b704a4eb680e8d8eb24c15dfd" candidate_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6e28b94711f499f993f8a61c6918bd8" is_pre_election: true
I20260620 14:45:57.187569 1734 leader_election.cc:302] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 033c799c2a984f309f921ed0344e79c3, 9587a2e3bdbd42b48e3ab68e8b8916a4; no voters:
I20260620 14:45:57.187676 1940 raft_consensus.cc:2454] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9587a2e3bdbd42b48e3ab68e8b8916a4 in term 0.
I20260620 14:45:57.187731 2014 raft_consensus.cc:2781] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:45:57.187830 2014 raft_consensus.cc:482] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:45:57.187867 2014 raft_consensus.cc:3037] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:57.188802 2014 raft_consensus.cc:504] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } } peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } }
I20260620 14:45:57.188910 2014 leader_election.cc:288] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [CANDIDATE]: Term 1 election: Requested vote from peers c6e28b94711f499f993f8a61c6918bd8 (127.0.98.3:34675), 033c799c2a984f309f921ed0344e79c3 (127.0.98.1:32885)
I20260620 14:45:57.189054 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d778b5b704a4eb680e8d8eb24c15dfd" candidate_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "033c799c2a984f309f921ed0344e79c3"
I20260620 14:45:57.189054 1940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d778b5b704a4eb680e8d8eb24c15dfd" candidate_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6e28b94711f499f993f8a61c6918bd8"
I20260620 14:45:57.189114 1661 raft_consensus.cc:3037] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:57.189119 1940 raft_consensus.cc:3037] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:57.189729 1661 raft_consensus.cc:2454] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9587a2e3bdbd42b48e3ab68e8b8916a4 in term 1.
I20260620 14:45:57.189806 1940 raft_consensus.cc:2454] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9587a2e3bdbd42b48e3ab68e8b8916a4 in term 1.
I20260620 14:45:57.189884 1734 leader_election.cc:302] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 033c799c2a984f309f921ed0344e79c3, 9587a2e3bdbd42b48e3ab68e8b8916a4; no voters:
I20260620 14:45:57.189961 2014 raft_consensus.cc:2781] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:45:57.190097 2014 raft_consensus.cc:686] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 1 LEADER]: Becoming Leader. State: Replica: 9587a2e3bdbd42b48e3ab68e8b8916a4, State: Running, Role: LEADER
I20260620 14:45:57.190212 2014 consensus_queue.cc:237] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } } peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } }
I20260620 14:45:57.190934 1503 catalog_manager.cc:5495] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9587a2e3bdbd42b48e3ab68e8b8916a4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } health_report { overall_health: HEALTHY } } }
I20260620 14:45:57.197901 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.195983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44506 (local address 127.0.98.2:42915)
0620 14:45:57.196018 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:57.196020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:57.196043 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:45:57.196111 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:57.196112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:57.196125 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:57.196178 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:45:57.196264 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.196684 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.197357 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.197523 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.197562 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.197688 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:45:57.197769 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:45:57.197817 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:57.197843 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 14:45:57.197855 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:45:57.199314 1661 raft_consensus.cc:1261] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3 [term 1 FOLLOWER]: Refusing update from remote peer 9587a2e3bdbd42b48e3ab68e8b8916a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:45:57.199319 1940 raft_consensus.cc:1261] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8 [term 1 FOLLOWER]: Refusing update from remote peer 9587a2e3bdbd42b48e3ab68e8b8916a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:45:57.199607 2014 consensus_queue.cc:1035] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:45:57.199729 2017 consensus_queue.cc:1035] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:45:57.201740 2020 mvcc.cc:205] Tried to move back new op lower bound from 7298935837486411776 to 7298935837451698176. Current Snapshot: MvccSnapshot[applied={T|T < 7298935837486411776}]
I20260620 14:45:57.202049 2022 mvcc.cc:205] Tried to move back new op lower bound from 7298935837486411776 to 7298935837451698176. Current Snapshot: MvccSnapshot[applied={T|T < 7298935837486411776}]
I20260620 14:45:57.204897 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:45:57.204995 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:45:57.205662 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b: Sending DeleteTablet for 3 replicas of tablet 8d778b5b704a4eb680e8d8eb24c15dfd
I20260620 14:45:57.205962 1640 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d778b5b704a4eb680e8d8eb24c15dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:45:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50876
I20260620 14:45:57.205960 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d778b5b704a4eb680e8d8eb24c15dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:45:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59214
I20260620 14:45:57.206158 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d778b5b704a4eb680e8d8eb24c15dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:45:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44498
I20260620 14:45:57.206190 2031 tablet_replica.cc:330] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8: stopping tablet replica
I20260620 14:45:57.206288 2031 raft_consensus.cc:2229] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:45:57.206388 2031 raft_consensus.cc:2258] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:45:57.206516 2032 tablet_replica.cc:330] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4: stopping tablet replica
I20260620 14:45:57.206458 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:45:57.206543 2030 tablet_replica.cc:330] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3: stopping tablet replica
W20260620 14:45:57.206596 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:45:57.206640 2030 raft_consensus.cc:2229] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:45:57.206734 2030 raft_consensus.cc:2258] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:45:57.206902 2032 raft_consensus.cc:2229] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:45:57.206930 2031 ts_tablet_manager.cc:1905] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:45:57.207024 2032 raft_consensus.cc:2258] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:45:57.207458 2032 ts_tablet_manager.cc:1905] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:45:57.207854 1781 tablet_service.cc:1467] Processing CreateTablet for tablet f92a938cbd4642958ecf4ac1637d91f5 (DEFAULT_TABLE table=test-table [id=7056310ab4794d56975d290b40f2cdbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:45:57.207968 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f92a938cbd4642958ecf4ac1637d91f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:57.208736 1921 tablet_service.cc:1467] Processing CreateTablet for tablet f92a938cbd4642958ecf4ac1637d91f5 (DEFAULT_TABLE table=test-table [id=7056310ab4794d56975d290b40f2cdbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:45:57.209295 1640 tablet_service.cc:1467] Processing CreateTablet for tablet f92a938cbd4642958ecf4ac1637d91f5 (DEFAULT_TABLE table=test-table [id=7056310ab4794d56975d290b40f2cdbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:45:57.209411 1640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f92a938cbd4642958ecf4ac1637d91f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:57.209450 2009 tablet_bootstrap.cc:492] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4: Bootstrap starting.
I20260620 14:45:57.209769 2009 tablet_bootstrap.cc:654] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:57.210152 2032 ts_tablet_manager.cc:1918] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:45:57.210314 2009 tablet_bootstrap.cc:492] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4: No bootstrap required, opened a new log
I20260620 14:45:57.210367 2032 log.cc:1196] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/8d778b5b704a4eb680e8d8eb24c15dfd
I20260620 14:45:57.210386 2009 ts_tablet_manager.cc:1397] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:45:57.210520 2009 raft_consensus.cc:348] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } }
I20260620 14:45:57.210577 2009 raft_consensus.cc:374] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:57.210597 2009 raft_consensus.cc:729] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9587a2e3bdbd42b48e3ab68e8b8916a4, State: Initialized, Role: FOLLOWER
I20260620 14:45:57.210624 2032 ts_tablet_manager.cc:1939] T 8d778b5b704a4eb680e8d8eb24c15dfd P 9587a2e3bdbd42b48e3ab68e8b8916a4: Deleting consensus metadata
I20260620 14:45:57.210638 2009 consensus_queue.cc:260] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } }
I20260620 14:45:57.210713 2009 ts_tablet_manager.cc:1428] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:57.210747 2010 tablet_bootstrap.cc:492] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3: Bootstrap starting.
I20260620 14:45:57.211107 2010 tablet_bootstrap.cc:654] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:57.211344 1489 catalog_manager.cc:4841] TS 9587a2e3bdbd42b48e3ab68e8b8916a4 (127.0.98.2:42915): tablet 8d778b5b704a4eb680e8d8eb24c15dfd (table test-table [id=e317168f3eca476f9841a5747f24e7cd]) successfully deleted
I20260620 14:45:57.211798 2030 ts_tablet_manager.cc:1905] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:45:57.213296 2030 ts_tablet_manager.cc:1918] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:45:57.213369 2030 log.cc:1196] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-0/wal/wals/8d778b5b704a4eb680e8d8eb24c15dfd
I20260620 14:45:57.213611 2030 ts_tablet_manager.cc:1939] T 8d778b5b704a4eb680e8d8eb24c15dfd P 033c799c2a984f309f921ed0344e79c3: Deleting consensus metadata
I20260620 14:45:57.214017 1490 catalog_manager.cc:4841] TS 033c799c2a984f309f921ed0344e79c3 (127.0.98.1:32885): tablet 8d778b5b704a4eb680e8d8eb24c15dfd (table test-table [id=e317168f3eca476f9841a5747f24e7cd]) successfully deleted
I20260620 14:45:57.214264 2010 tablet_bootstrap.cc:492] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3: No bootstrap required, opened a new log
I20260620 14:45:57.214306 2010 ts_tablet_manager.cc:1397] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 14:45:57.214452 2010 raft_consensus.cc:348] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } }
I20260620 14:45:57.214504 2010 raft_consensus.cc:374] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:57.214520 2010 raft_consensus.cc:729] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 033c799c2a984f309f921ed0344e79c3, State: Initialized, Role: FOLLOWER
I20260620 14:45:57.214560 2010 consensus_queue.cc:260] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } }
I20260620 14:45:57.214646 2010 ts_tablet_manager.cc:1428] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:57.215557 2031 ts_tablet_manager.cc:1918] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:45:57.215583 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f92a938cbd4642958ecf4ac1637d91f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:57.215626 2031 log.cc:1196] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/8d778b5b704a4eb680e8d8eb24c15dfd
I20260620 14:45:57.215830 2031 ts_tablet_manager.cc:1939] T 8d778b5b704a4eb680e8d8eb24c15dfd P c6e28b94711f499f993f8a61c6918bd8: Deleting consensus metadata
I20260620 14:45:57.216765 2007 tablet_bootstrap.cc:492] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8: Bootstrap starting.
I20260620 14:45:57.216858 1488 catalog_manager.cc:4841] TS c6e28b94711f499f993f8a61c6918bd8 (127.0.98.3:34675): tablet 8d778b5b704a4eb680e8d8eb24c15dfd (table test-table [id=e317168f3eca476f9841a5747f24e7cd]) successfully deleted
I20260620 14:45:57.217201 2007 tablet_bootstrap.cc:654] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:57.217710 2007 tablet_bootstrap.cc:492] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8: No bootstrap required, opened a new log
I20260620 14:45:57.217749 2007 ts_tablet_manager.cc:1397] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:45:57.217876 2007 raft_consensus.cc:348] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } }
I20260620 14:45:57.217931 2007 raft_consensus.cc:374] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:57.217947 2007 raft_consensus.cc:729] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6e28b94711f499f993f8a61c6918bd8, State: Initialized, Role: FOLLOWER
I20260620 14:45:57.217989 2007 consensus_queue.cc:260] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } }
I20260620 14:45:57.218072 2007 ts_tablet_manager.cc:1428] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260620 14:45:57.219527 1848 tablet.cc:2369] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:45:57.247663 2015 raft_consensus.cc:482] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:45:57.247750 2015 raft_consensus.cc:504] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } }
I20260620 14:45:57.248077 2015 leader_election.cc:288] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9587a2e3bdbd42b48e3ab68e8b8916a4 (127.0.98.2:42915), c6e28b94711f499f993f8a61c6918bd8 (127.0.98.3:34675)
I20260620 14:45:57.251220 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.248229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42915 (local address 127.0.98.1:35269)
0620 14:45:57.248290 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:57.248301 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:57.248339 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:57.248490 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:57.248491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:57.248493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:45:57.248559 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.248561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.249194 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.249195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.250247 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.250250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.251012 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.251013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.251078 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.251080 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:45:57.251121 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 14:45:57.251154 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:45:57.251242 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.248308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35269 (local address 127.0.98.2:42915)
0620 14:45:57.248348 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:57.248350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:57.248361 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:45:57.248388 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:57.248388 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:57.248397 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:57.248442 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:45:57.248627 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.249123 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.250340 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.250926 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.250998 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.251009 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:57.251138 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 14:45:57.251173 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:45:57.251581 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.248423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44667 (local address 127.0.98.3:34675)
0620 14:45:57.248476 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:57.248478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:57.248508 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:45:57.248534 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:57.248534 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:57.248543 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:57.248589 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:45:57.248739 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.249455 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.250590 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.250936 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.251001 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.251011 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:57.251478 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 14:45:57.251523 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:45:57.251570 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92a938cbd4642958ecf4ac1637d91f5" candidate_uuid: "033c799c2a984f309f921ed0344e79c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" is_pre_election: true
I20260620 14:45:57.251691 1801 raft_consensus.cc:2454] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 033c799c2a984f309f921ed0344e79c3 in term 0.
I20260620 14:45:57.251760 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.248241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34675 (local address 127.0.98.1:44667)
0620 14:45:57.248432 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:57.248440 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:57.248474 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:57.248607 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:57.248608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:57.248609 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:45:57.248677 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.248678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.249555 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.249556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.250515 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.250517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.251307 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.251307 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.251371 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.251374 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:45:57.251650 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 14:45:57.251694 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":84,"threads_started":1}
I20260620 14:45:57.251875 1593 leader_election.cc:302] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 033c799c2a984f309f921ed0344e79c3, 9587a2e3bdbd42b48e3ab68e8b8916a4; no voters:
I20260620 14:45:57.251993 2015 raft_consensus.cc:2781] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:45:57.252023 2015 raft_consensus.cc:482] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:45:57.252038 2015 raft_consensus.cc:3037] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:57.252002 1940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92a938cbd4642958ecf4ac1637d91f5" candidate_uuid: "033c799c2a984f309f921ed0344e79c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6e28b94711f499f993f8a61c6918bd8" is_pre_election: true
I20260620 14:45:57.252079 1940 raft_consensus.cc:2454] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 033c799c2a984f309f921ed0344e79c3 in term 0.
I20260620 14:45:57.252712 2015 raft_consensus.cc:504] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } }
I20260620 14:45:57.252825 2015 leader_election.cc:288] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [CANDIDATE]: Term 1 election: Requested vote from peers 9587a2e3bdbd42b48e3ab68e8b8916a4 (127.0.98.2:42915), c6e28b94711f499f993f8a61c6918bd8 (127.0.98.3:34675)
I20260620 14:45:57.252977 1940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92a938cbd4642958ecf4ac1637d91f5" candidate_uuid: "033c799c2a984f309f921ed0344e79c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6e28b94711f499f993f8a61c6918bd8"
I20260620 14:45:57.252985 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92a938cbd4642958ecf4ac1637d91f5" candidate_uuid: "033c799c2a984f309f921ed0344e79c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4"
I20260620 14:45:57.253046 1801 raft_consensus.cc:3037] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:57.253046 1940 raft_consensus.cc:3037] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:57.253752 1801 raft_consensus.cc:2454] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 033c799c2a984f309f921ed0344e79c3 in term 1.
I20260620 14:45:57.253803 1940 raft_consensus.cc:2454] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 033c799c2a984f309f921ed0344e79c3 in term 1.
I20260620 14:45:57.253911 1593 leader_election.cc:302] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 033c799c2a984f309f921ed0344e79c3, 9587a2e3bdbd42b48e3ab68e8b8916a4; no voters:
I20260620 14:45:57.254012 2015 raft_consensus.cc:2781] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:45:57.254165 2015 raft_consensus.cc:686] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [term 1 LEADER]: Becoming Leader. State: Replica: 033c799c2a984f309f921ed0344e79c3, State: Running, Role: LEADER
I20260620 14:45:57.254233 2015 consensus_queue.cc:237] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } }
I20260620 14:45:57.254833 1503 catalog_manager.cc:5495] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 033c799c2a984f309f921ed0344e79c3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "033c799c2a984f309f921ed0344e79c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:45:57 dist-test-slave-cx89 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:45:57.291497 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.284177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54568 (local address 127.0.98.62:44051)
0620 14:45:57.284232 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:57.284234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:57.284342 (+ 108us) server_negotiation.cc:407] Connection header received
0620 14:45:57.284373 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:57.284374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:57.284383 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:57.284442 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:57.284990 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.285514 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.286529 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.286703 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.286744 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.289062 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:57.289063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:57.289063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:57.289070 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:57.290415 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:57.290755 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:57.290756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:57.290756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:57.290814 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:57.290983 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:57.290984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:57.290984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:57.291093 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:57.291147 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:57.291272 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:45:57.291356 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":34}
I20260620 14:45:57.291556 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.284074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44051 (local address 127.0.0.1:54568)
0620 14:45:57.284232 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:57.284246 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:57.284328 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:57.284451 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:57.284452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:57.284858 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:57.284916 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.284918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.285602 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.285603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.286435 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.286438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.286772 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.286772 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.286832 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.287401 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:57.287406 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:57.288981 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:57.290505 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:57.290511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:57.290513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:57.290692 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:57.290870 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:57.290870 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:57.290870 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:57.290926 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:57.291144 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:57.291150 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:57.291192 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:45:57.291245 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:45:57.291429 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":64,"thread_start_us":84,"threads_started":1}
I20260620 14:45:57.293387 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54568:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:45:57.293642 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:45:57.294902 1640 tablet_service.cc:1467] Processing CreateTablet for tablet b842ab89cef545ce85e17439213699a6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a751381bdaf9453699526efc4203550c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:45:57.294919 1781 tablet_service.cc:1467] Processing CreateTablet for tablet b842ab89cef545ce85e17439213699a6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a751381bdaf9453699526efc4203550c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:45:57.295055 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b842ab89cef545ce85e17439213699a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:57.295056 1640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b842ab89cef545ce85e17439213699a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:57.296207 2009 tablet_bootstrap.cc:492] T b842ab89cef545ce85e17439213699a6 P 9587a2e3bdbd42b48e3ab68e8b8916a4: Bootstrap starting.
I20260620 14:45:57.296509 2010 tablet_bootstrap.cc:492] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3: Bootstrap starting.
I20260620 14:45:57.296597 2009 tablet_bootstrap.cc:654] T b842ab89cef545ce85e17439213699a6 P 9587a2e3bdbd42b48e3ab68e8b8916a4: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:57.296878 2010 tablet_bootstrap.cc:654] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:57.297319 2009 tablet_bootstrap.cc:492] T b842ab89cef545ce85e17439213699a6 P 9587a2e3bdbd42b48e3ab68e8b8916a4: No bootstrap required, opened a new log
I20260620 14:45:57.297360 2009 ts_tablet_manager.cc:1397] T b842ab89cef545ce85e17439213699a6 P 9587a2e3bdbd42b48e3ab68e8b8916a4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:45:57.297327 1921 tablet_service.cc:1467] Processing CreateTablet for tablet b842ab89cef545ce85e17439213699a6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a751381bdaf9453699526efc4203550c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:45:57.297448 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b842ab89cef545ce85e17439213699a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:57.297510 2009 raft_consensus.cc:348] T b842ab89cef545ce85e17439213699a6 P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } }
I20260620 14:45:57.297583 2009 raft_consensus.cc:374] T b842ab89cef545ce85e17439213699a6 P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:57.297601 2009 raft_consensus.cc:729] T b842ab89cef545ce85e17439213699a6 P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9587a2e3bdbd42b48e3ab68e8b8916a4, State: Initialized, Role: FOLLOWER
I20260620 14:45:57.297641 2009 consensus_queue.cc:260] T b842ab89cef545ce85e17439213699a6 P 9587a2e3bdbd42b48e3ab68e8b8916a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } }
I20260620 14:45:57.297729 2009 ts_tablet_manager.cc:1428] T b842ab89cef545ce85e17439213699a6 P 9587a2e3bdbd42b48e3ab68e8b8916a4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 14:45:57.297824 2017 tablet_replica.cc:439] T b842ab89cef545ce85e17439213699a6 P 9587a2e3bdbd42b48e3ab68e8b8916a4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } } }
I20260620 14:45:57.297881 2017 tablet_replica.cc:442] T b842ab89cef545ce85e17439213699a6 P 9587a2e3bdbd42b48e3ab68e8b8916a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:45:57.298826 2007 tablet_bootstrap.cc:492] T b842ab89cef545ce85e17439213699a6 P c6e28b94711f499f993f8a61c6918bd8: Bootstrap starting.
I20260620 14:45:57.299069 2010 tablet_bootstrap.cc:492] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3: No bootstrap required, opened a new log
I20260620 14:45:57.299124 2010 ts_tablet_manager.cc:1397] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:45:57.299249 2010 raft_consensus.cc:348] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } }
I20260620 14:45:57.299319 2010 raft_consensus.cc:374] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:57.299337 2010 raft_consensus.cc:729] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 033c799c2a984f309f921ed0344e79c3, State: Initialized, Role: FOLLOWER
I20260620 14:45:57.299372 2010 consensus_queue.cc:260] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } }
I20260620 14:45:57.299481 2010 ts_tablet_manager.cc:1428] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 14:45:57.299610 2015 tablet_replica.cc:439] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } } }
I20260620 14:45:57.299655 2007 tablet_bootstrap.cc:654] T b842ab89cef545ce85e17439213699a6 P c6e28b94711f499f993f8a61c6918bd8: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:57.299675 2015 tablet_replica.cc:442] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:45:57.300194 2007 tablet_bootstrap.cc:492] T b842ab89cef545ce85e17439213699a6 P c6e28b94711f499f993f8a61c6918bd8: No bootstrap required, opened a new log
I20260620 14:45:57.300242 2007 ts_tablet_manager.cc:1397] T b842ab89cef545ce85e17439213699a6 P c6e28b94711f499f993f8a61c6918bd8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:45:57.300360 2007 raft_consensus.cc:348] T b842ab89cef545ce85e17439213699a6 P c6e28b94711f499f993f8a61c6918bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } }
I20260620 14:45:57.300418 2007 raft_consensus.cc:374] T b842ab89cef545ce85e17439213699a6 P c6e28b94711f499f993f8a61c6918bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:57.300431 2007 raft_consensus.cc:729] T b842ab89cef545ce85e17439213699a6 P c6e28b94711f499f993f8a61c6918bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6e28b94711f499f993f8a61c6918bd8, State: Initialized, Role: FOLLOWER
I20260620 14:45:57.300462 2007 consensus_queue.cc:260] T b842ab89cef545ce85e17439213699a6 P c6e28b94711f499f993f8a61c6918bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } }
I20260620 14:45:57.300540 2007 ts_tablet_manager.cc:1428] T b842ab89cef545ce85e17439213699a6 P c6e28b94711f499f993f8a61c6918bd8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:57.300596 2011 tablet_replica.cc:439] T b842ab89cef545ce85e17439213699a6 P c6e28b94711f499f993f8a61c6918bd8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } } }
I20260620 14:45:57.300675 2011 tablet_replica.cc:442] T b842ab89cef545ce85e17439213699a6 P c6e28b94711f499f993f8a61c6918bd8: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 14:45:57.311484 1708 tablet.cc:2369] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:45:57.326789 2015 raft_consensus.cc:482] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:45:57.326879 2015 raft_consensus.cc:504] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } }
I20260620 14:45:57.327028 2015 leader_election.cc:288] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9587a2e3bdbd42b48e3ab68e8b8916a4 (127.0.98.2:42915), c6e28b94711f499f993f8a61c6918bd8 (127.0.98.3:34675)
I20260620 14:45:57.327207 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b842ab89cef545ce85e17439213699a6" candidate_uuid: "033c799c2a984f309f921ed0344e79c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" is_pre_election: true
I20260620 14:45:57.327207 1940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b842ab89cef545ce85e17439213699a6" candidate_uuid: "033c799c2a984f309f921ed0344e79c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6e28b94711f499f993f8a61c6918bd8" is_pre_election: true
I20260620 14:45:57.327309 1940 raft_consensus.cc:2454] T b842ab89cef545ce85e17439213699a6 P c6e28b94711f499f993f8a61c6918bd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 033c799c2a984f309f921ed0344e79c3 in term 0.
I20260620 14:45:57.327309 1801 raft_consensus.cc:2454] T b842ab89cef545ce85e17439213699a6 P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 033c799c2a984f309f921ed0344e79c3 in term 0.
I20260620 14:45:57.327477 1593 leader_election.cc:302] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 033c799c2a984f309f921ed0344e79c3, 9587a2e3bdbd42b48e3ab68e8b8916a4; no voters:
I20260620 14:45:57.327584 2015 raft_consensus.cc:2781] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:45:57.327634 2015 raft_consensus.cc:482] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:45:57.327654 2015 raft_consensus.cc:3037] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:57.328426 2015 raft_consensus.cc:504] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } }
I20260620 14:45:57.328547 2015 leader_election.cc:288] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 [CANDIDATE]: Term 1 election: Requested vote from peers 9587a2e3bdbd42b48e3ab68e8b8916a4 (127.0.98.2:42915), c6e28b94711f499f993f8a61c6918bd8 (127.0.98.3:34675)
I20260620 14:45:57.328670 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b842ab89cef545ce85e17439213699a6" candidate_uuid: "033c799c2a984f309f921ed0344e79c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4"
I20260620 14:45:57.328701 1940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b842ab89cef545ce85e17439213699a6" candidate_uuid: "033c799c2a984f309f921ed0344e79c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6e28b94711f499f993f8a61c6918bd8"
I20260620 14:45:57.328742 1801 raft_consensus.cc:3037] T b842ab89cef545ce85e17439213699a6 P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:57.328763 1940 raft_consensus.cc:3037] T b842ab89cef545ce85e17439213699a6 P c6e28b94711f499f993f8a61c6918bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:57.329282 1940 raft_consensus.cc:2454] T b842ab89cef545ce85e17439213699a6 P c6e28b94711f499f993f8a61c6918bd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 033c799c2a984f309f921ed0344e79c3 in term 1.
I20260620 14:45:57.329308 1801 raft_consensus.cc:2454] T b842ab89cef545ce85e17439213699a6 P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 033c799c2a984f309f921ed0344e79c3 in term 1.
I20260620 14:45:57.329452 1592 leader_election.cc:302] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 033c799c2a984f309f921ed0344e79c3, c6e28b94711f499f993f8a61c6918bd8; no voters:
I20260620 14:45:57.329532 2015 raft_consensus.cc:2781] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:45:57.329581 2015 raft_consensus.cc:686] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 [term 1 LEADER]: Becoming Leader. State: Replica: 033c799c2a984f309f921ed0344e79c3, State: Running, Role: LEADER
I20260620 14:45:57.329628 2015 consensus_queue.cc:237] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } }
I20260620 14:45:57.329923 2037 tablet_replica.cc:439] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3: TxnStatusTablet state changed. Reason: New leader 033c799c2a984f309f921ed0344e79c3. Latest consensus state: current_term: 1 leader_uuid: "033c799c2a984f309f921ed0344e79c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } } }
I20260620 14:45:57.329998 2037 tablet_replica.cc:442] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:45:57.330124 2049 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:45:57.330233 2049 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:45:57.330189 1503 catalog_manager.cc:5495] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 033c799c2a984f309f921ed0344e79c3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "033c799c2a984f309f921ed0344e79c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } health_report { overall_health: UNKNOWN } } }
I20260620 14:45:57.340090 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.337695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32885 (local address 127.0.0.1:50884)
0620 14:45:57.337760 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:57.337772 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:57.337820 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:57.338003 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:57.338004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:57.338005 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:45:57.338065 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.338068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.338657 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.338658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.339215 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.339217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.339607 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.339609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.339669 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.339682 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:45:57.339917 (+ 235us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:45:57.339925 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:45:57.339974 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:45:57.340025 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 14:45:57.340075 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.337748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50884 (local address 127.0.98.1:32885)
0620 14:45:57.337894 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:57.337896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:57.337901 (+ 5us) server_negotiation.cc:407] Connection header received
0620 14:45:57.337926 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:57.337926 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:57.337941 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:57.337988 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:45:57.338122 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.338592 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.339287 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.339541 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.339592 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.339758 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:45:57.339858 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:45:57.339925 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:57.339978 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 14:45:57.340002 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260620 14:45:57.341137 1940 raft_consensus.cc:1261] T b842ab89cef545ce85e17439213699a6 P c6e28b94711f499f993f8a61c6918bd8 [term 1 FOLLOWER]: Refusing update from remote peer 033c799c2a984f309f921ed0344e79c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:45:57.341154 1801 raft_consensus.cc:1261] T b842ab89cef545ce85e17439213699a6 P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 1 FOLLOWER]: Refusing update from remote peer 033c799c2a984f309f921ed0344e79c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:45:57.341357 2037 consensus_queue.cc:1035] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:45:57.341476 2015 consensus_queue.cc:1035] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:45:57.342025 2011 tablet_replica.cc:439] T b842ab89cef545ce85e17439213699a6 P c6e28b94711f499f993f8a61c6918bd8: TxnStatusTablet state changed. Reason: New leader 033c799c2a984f309f921ed0344e79c3. Latest consensus state: current_term: 1 leader_uuid: "033c799c2a984f309f921ed0344e79c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } } }
I20260620 14:45:57.342116 2011 tablet_replica.cc:442] T b842ab89cef545ce85e17439213699a6 P c6e28b94711f499f993f8a61c6918bd8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:45:57.342072 2017 tablet_replica.cc:439] T b842ab89cef545ce85e17439213699a6 P 9587a2e3bdbd42b48e3ab68e8b8916a4: TxnStatusTablet state changed. Reason: New leader 033c799c2a984f309f921ed0344e79c3. Latest consensus state: current_term: 1 leader_uuid: "033c799c2a984f309f921ed0344e79c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } } }
I20260620 14:45:57.342160 2017 tablet_replica.cc:442] T b842ab89cef545ce85e17439213699a6 P 9587a2e3bdbd42b48e3ab68e8b8916a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:45:57.342854 2015 tablet_replica.cc:439] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "033c799c2a984f309f921ed0344e79c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } } }
I20260620 14:45:57.342873 2037 tablet_replica.cc:439] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "033c799c2a984f309f921ed0344e79c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } } }
I20260620 14:45:57.342936 2015 tablet_replica.cc:442] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:45:57.342947 2037 tablet_replica.cc:442] T b842ab89cef545ce85e17439213699a6 P 033c799c2a984f309f921ed0344e79c3: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:45:57.343537 2017 tablet_replica.cc:439] T b842ab89cef545ce85e17439213699a6 P 9587a2e3bdbd42b48e3ab68e8b8916a4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "033c799c2a984f309f921ed0344e79c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } } }
I20260620 14:45:57.343612 2017 tablet_replica.cc:442] T b842ab89cef545ce85e17439213699a6 P 9587a2e3bdbd42b48e3ab68e8b8916a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:45:57.344035 2011 tablet_replica.cc:439] T b842ab89cef545ce85e17439213699a6 P c6e28b94711f499f993f8a61c6918bd8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "033c799c2a984f309f921ed0344e79c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "033c799c2a984f309f921ed0344e79c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32885 } } peers { permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 } } peers { permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 } } }
I20260620 14:45:57.344110 2011 tablet_replica.cc:442] T b842ab89cef545ce85e17439213699a6 P c6e28b94711f499f993f8a61c6918bd8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:45:57.346746 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.344809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50886 (local address 127.0.98.1:32885)
0620 14:45:57.344854 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:57.344855 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:57.344861 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:45:57.344879 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:57.344880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:57.344902 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:57.344944 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:45:57.345058 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.345504 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.346173 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.346364 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.346420 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.346538 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:45:57.346569 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:45:57.346610 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:57.346665 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 14:45:57.346682 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:45:57.351372 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.348893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60095 (local address 127.0.98.1:32885)
0620 14:45:57.348931 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:57.348932 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:57.349018 (+ 86us) server_negotiation.cc:407] Connection header received
0620 14:45:57.349084 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:57.349085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:57.349091 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:57.349129 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:45:57.349249 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.349665 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.350767 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.350943 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.350994 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.351154 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:45:57.351175 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:45:57.351202 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:57.351269 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 14:45:57.351301 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":22}
I20260620 14:45:57.351483 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.348825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32885 (local address 127.0.98.1:60095)
0620 14:45:57.348957 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:57.348974 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:57.349030 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:57.349134 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:57.349135 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:57.349136 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:45:57.349190 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.349191 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.349721 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.349722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.350687 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:57.350689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.351007 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.351008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:57.351069 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.351084 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:45:57.351217 (+ 133us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:45:57.351218 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:45:57.351266 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:45:57.351321 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":3,"thread_start_us":63,"threads_started":1}
I20260620 14:45:57.353267 1801 raft_consensus.cc:1261] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 1 FOLLOWER]: Refusing update from remote peer 033c799c2a984f309f921ed0344e79c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:45:57.353262 1941 raft_consensus.cc:1261] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8 [term 1 FOLLOWER]: Refusing update from remote peer 033c799c2a984f309f921ed0344e79c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:45:57.353437 2015 consensus_queue.cc:1035] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9587a2e3bdbd42b48e3ab68e8b8916a4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:45:57.353523 2037 consensus_queue.cc:1035] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6e28b94711f499f993f8a61c6918bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:45:57.364372 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:45:57.364465 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:45:57.364998 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e681cf6156da467d933f61e14969fb9b: Sending DeleteTablet for 3 replicas of tablet f92a938cbd4642958ecf4ac1637d91f5
I20260620 14:45:57.365221 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet f92a938cbd4642958ecf4ac1637d91f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:45:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59214
I20260620 14:45:57.365281 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet f92a938cbd4642958ecf4ac1637d91f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:45:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44498
I20260620 14:45:57.365311 2031 tablet_replica.cc:330] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8: stopping tablet replica
I20260620 14:45:57.365351 2031 raft_consensus.cc:2229] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:45:57.365365 2032 tablet_replica.cc:330] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4: stopping tablet replica
I20260620 14:45:57.365382 2031 raft_consensus.cc:2258] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:45:57.365406 2032 raft_consensus.cc:2229] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:45:57.365435 2032 raft_consensus.cc:2258] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:45:57.365620 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet f92a938cbd4642958ecf4ac1637d91f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:45:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50876
I20260620 14:45:57.365702 2030 tablet_replica.cc:330] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3: stopping tablet replica
I20260620 14:45:57.365754 2030 raft_consensus.cc:2229] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:45:57.365828 2030 raft_consensus.cc:2258] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:45:57.365931 2032 ts_tablet_manager.cc:1905] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:45:57.366092 2030 ts_tablet_manager.cc:1905] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:45:57.367314 2031 ts_tablet_manager.cc:1905] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:45:57.367515 2032 ts_tablet_manager.cc:1918] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:45:57.367539 2030 ts_tablet_manager.cc:1918] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:45:57.367568 2032 log.cc:1196] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/f92a938cbd4642958ecf4ac1637d91f5
I20260620 14:45:57.367595 2030 log.cc:1196] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-0/wal/wals/f92a938cbd4642958ecf4ac1637d91f5
I20260620 14:45:57.367774 2032 ts_tablet_manager.cc:1939] T f92a938cbd4642958ecf4ac1637d91f5 P 9587a2e3bdbd42b48e3ab68e8b8916a4: Deleting consensus metadata
I20260620 14:45:57.367803 2030 ts_tablet_manager.cc:1939] T f92a938cbd4642958ecf4ac1637d91f5 P 033c799c2a984f309f921ed0344e79c3: Deleting consensus metadata
I20260620 14:45:57.368031 1489 catalog_manager.cc:4841] TS 9587a2e3bdbd42b48e3ab68e8b8916a4 (127.0.98.2:42915): tablet f92a938cbd4642958ecf4ac1637d91f5 (table test-table [id=7056310ab4794d56975d290b40f2cdbe]) successfully deleted
I20260620 14:45:57.368096 1490 catalog_manager.cc:4841] TS 033c799c2a984f309f921ed0344e79c3 (127.0.98.1:32885): tablet f92a938cbd4642958ecf4ac1637d91f5 (table test-table [id=7056310ab4794d56975d290b40f2cdbe]) successfully deleted
I20260620 14:45:57.368741 2031 ts_tablet_manager.cc:1918] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:45:57.368801 2031 log.cc:1196] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/f92a938cbd4642958ecf4ac1637d91f5
I20260620 14:45:57.368989 2031 ts_tablet_manager.cc:1939] T f92a938cbd4642958ecf4ac1637d91f5 P c6e28b94711f499f993f8a61c6918bd8: Deleting consensus metadata
I20260620 14:45:57.369441 1488 catalog_manager.cc:4841] TS c6e28b94711f499f993f8a61c6918bd8 (127.0.98.3:34675): tablet f92a938cbd4642958ecf4ac1637d91f5 (table test-table [id=7056310ab4794d56975d290b40f2cdbe]) successfully deleted
Jun 20 14:45:57 dist-test-slave-cx89 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:45:57.382891 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.367130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.0.98.1:32885)
0620 14:45:57.367171 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:57.367173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:57.375216 (+ 8043us) server_negotiation.cc:407] Connection header received
0620 14:45:57.375275 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:57.375276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:57.375286 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:57.375329 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:57.375900 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.376362 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.377040 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.377218 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.377269 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.380251 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:57.380259 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:57.380259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:57.380266 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:57.381824 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:57.382215 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:57.382218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:57.382218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:57.382273 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:57.382442 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:57.382443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:57.382443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:57.382573 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:57.382706 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:57.382747 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 14:45:57.382809 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 14:45:57.383106 1701 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50890
I20260620 14:45:57.389912 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:57.384043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54582 (local address 127.0.98.62:44051)
0620 14:45:57.384100 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:57.384102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:57.384145 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:45:57.384192 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:57.384193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:57.384214 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:57.384295 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:57.384712 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.385478 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.386114 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:57.386414 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:57.386481 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:57.387581 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:57.387582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:57.387583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:57.387590 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:57.388943 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:57.389230 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:57.389231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:57.389231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:57.389292 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:57.389458 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:57.389458 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:57.389459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:57.389588 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:57.389676 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:57.389745 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:45:57.389818 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 14:45:57.390142 1503 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54582
I20260620 14:45:57.390796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 1576
I20260620 14:45:57.396651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 1716
I20260620 14:45:57.401384 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 1856
I20260620 14:45:57.404974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 1472
2026-06-20T14:45:57Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (1956 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:45:57 dist-test-slave-cx89 krb5kdc[2067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:45:57 dist-test-slave-cx89 krb5kdc[2067](info): set up 2 sockets
Jun 20 14:45:57 dist-test-slave-cx89 krb5kdc[2067](info): commencing operation
krb5kdc: starting...
W20260620 14:45:58.346830 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:45:58 dist-test-slave-cx89 krb5kdc[2067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:45:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:45:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:45:58.457783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33065
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:45:58.535674 2080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:45:58.535874 2080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:45:58.535895 2080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:45:58.537060 2080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:45:58.537101 2080 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:45:58.537113 2080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:45:58.537124 2080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:45:58.537135 2080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:45:58.537146 2080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:45:58.538585 2080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33065
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:45:58.538810 2080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:45:58.539031 2080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:45:58.543457 2088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:58.543468 2086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:58.543468 2085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:58.543653 2080 server_base.cc:1034] running on GCE node
I20260620 14:45:58.543861 2080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:45:58.544114 2080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:45:58.545248 2080 hybrid_clock.cc:648] HybridClock initialized: now 1781966758545236 us; error 29 us; skew 500 ppm
Jun 20 14:45:58 dist-test-slave-cx89 krb5kdc[2067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:45:58.547564 2080 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:45:58.547996 2080 webserver.cc:460] Webserver started at http://127.0.98.62:37483/ using document root <none> and password file <none>
I20260620 14:45:58.548192 2080 fs_manager.cc:359] Metadata directory not provided
I20260620 14:45:58.548238 2080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:45:58.548360 2080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:45:58.549362 2080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "3bab7abae4d7447097cc15d754bb9277"
format_stamp: "Formatted at 2026-06-20 14:45:58 on dist-test-slave-cx89"
server_key: "6ebb18dc90876d569107e6cf9dc8c9bb"
server_key_iv: "e46096d4ccd3a9485faa1cfd86ba59e7"
server_key_version: "encryptionkey@0"
I20260620 14:45:58.549693 2080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "3bab7abae4d7447097cc15d754bb9277"
format_stamp: "Formatted at 2026-06-20 14:45:58 on dist-test-slave-cx89"
server_key: "6ebb18dc90876d569107e6cf9dc8c9bb"
server_key_iv: "e46096d4ccd3a9485faa1cfd86ba59e7"
server_key_version: "encryptionkey@0"
I20260620 14:45:58.550943 2080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:45:58.551676 2095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:58.551826 2080 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:45:58.551918 2080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "3bab7abae4d7447097cc15d754bb9277"
format_stamp: "Formatted at 2026-06-20 14:45:58 on dist-test-slave-cx89"
server_key: "6ebb18dc90876d569107e6cf9dc8c9bb"
server_key_iv: "e46096d4ccd3a9485faa1cfd86ba59e7"
server_key_version: "encryptionkey@0"
I20260620 14:45:58.551985 2080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:45:58.567226 2080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:45:58.569439 2080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:45:58.569581 2080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:45:58.574096 2080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33065
I20260620 14:45:58.574158 2157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33065 every 8 connection(s)
I20260620 14:45:58.574463 2080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:45:58.575035 2158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:58.577230 2158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277: Bootstrap starting.
I20260620 14:45:58.577776 2158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:58.578032 2158 log.cc:824] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277: Log is configured to *not* fsync() on all Append() calls
I20260620 14:45:58.578655 2158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277: No bootstrap required, opened a new log
I20260620 14:45:58.580682 2158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bab7abae4d7447097cc15d754bb9277" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } }
I20260620 14:45:58.580823 2158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:58.580844 2158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bab7abae4d7447097cc15d754bb9277, State: Initialized, Role: FOLLOWER
I20260620 14:45:58.580977 2158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bab7abae4d7447097cc15d754bb9277" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } }
I20260620 14:45:58.581045 2158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:45:58.581077 2158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:45:58.581121 2158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:58.581607 2158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bab7abae4d7447097cc15d754bb9277" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } }
I20260620 14:45:58.581692 2158 leader_election.cc:302] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bab7abae4d7447097cc15d754bb9277; no voters:
I20260620 14:45:58.581837 2158 leader_election.cc:288] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:45:58.581898 2161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:45:58.582029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 2080
I20260620 14:45:58.582063 2161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277 [term 1 LEADER]: Becoming Leader. State: Replica: 3bab7abae4d7447097cc15d754bb9277, State: Running, Role: LEADER
I20260620 14:45:58.582111 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:45:58.582103 2158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:45:58.582152 2161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bab7abae4d7447097cc15d754bb9277" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } }
I20260620 14:45:58.582276 392 external_mini_cluster.cc:1428] Setting key 449132f6baad477cbb2dcce5b7e2e391
I20260620 14:45:58.582530 2162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bab7abae4d7447097cc15d754bb9277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bab7abae4d7447097cc15d754bb9277" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } } }
I20260620 14:45:58.582605 2162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277 [sys.catalog]: This master's current role is: LEADER
I20260620 14:45:58.582746 2163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bab7abae4d7447097cc15d754bb9277. Latest consensus state: current_term: 1 leader_uuid: "3bab7abae4d7447097cc15d754bb9277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bab7abae4d7447097cc15d754bb9277" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33065 } } }
I20260620 14:45:58.582792 2163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277 [sys.catalog]: This master's current role is: LEADER
I20260620 14:45:58.582935 2170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:45:58.583482 2170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:45:58.585080 2170 catalog_manager.cc:1347] Generated new cluster ID: 0f8ce5ff0e6e4205ad52644037aafdb9
I20260620 14:45:58.585131 2170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:45:58 dist-test-slave-cx89 krb5kdc[2067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:45:58.594048 2170 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:45:58.594271 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:58.582724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51754 (local address 127.0.98.62:33065)
0620 14:45:58.583687 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:58.583696 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:58.583872 (+ 176us) server_negotiation.cc:407] Connection header received
0620 14:45:58.584120 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:58.584129 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:58.584183 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:58.584304 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:58.585056 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:58.586442 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:58.587504 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:58.588149 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:58.588264 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:58.590722 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:58.590726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:58.590726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:58.590738 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:58.592600 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:58.592949 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:58.592952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:58.592953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:58.593061 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:58.593304 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:58.593304 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:58.593305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:58.593601 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:58.593747 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:58.593852 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 14:45:58.593997 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":57,"threads_started":1}
I20260620 14:45:58.594696 2170 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:45:58.601395 2170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277: Generated new TSK 0
I20260620 14:45:58.601574 2170 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:45:58.626498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33065
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:45:58.705174 2184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:45:58.705390 2184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:45:58.705415 2184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:45:58.706591 2184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:45:58.706638 2184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:45:58.706658 2184 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:45:58.706669 2184 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:45:58.706686 2184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:45:58.708156 2184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:45:58.708381 2184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:45:58.708611 2184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:45:58.713322 2192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:58.713451 2190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:58.713361 2184 server_base.cc:1034] running on GCE node
W20260620 14:45:58.713474 2189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:58.713721 2184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:45:58.713944 2184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:45:58.715077 2184 hybrid_clock.cc:648] HybridClock initialized: now 1781966758715062 us; error 28 us; skew 500 ppm
Jun 20 14:45:58 dist-test-slave-cx89 krb5kdc[2067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:45:58.717382 2184 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:45:58.717756 2184 webserver.cc:460] Webserver started at http://127.0.98.1:36685/ using document root <none> and password file <none>
I20260620 14:45:58.717933 2184 fs_manager.cc:359] Metadata directory not provided
I20260620 14:45:58.717970 2184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:45:58.718062 2184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:45:58.718988 2184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "83c0797ece354bcda365f0fc36cb476e"
format_stamp: "Formatted at 2026-06-20 14:45:58 on dist-test-slave-cx89"
server_key: "64f263641750f829dc6c2f34af1957ee"
server_key_iv: "f96aaa4c58a07cff84fbfd5d875e2448"
server_key_version: "encryptionkey@0"
I20260620 14:45:58.719272 2184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "83c0797ece354bcda365f0fc36cb476e"
format_stamp: "Formatted at 2026-06-20 14:45:58 on dist-test-slave-cx89"
server_key: "64f263641750f829dc6c2f34af1957ee"
server_key_iv: "f96aaa4c58a07cff84fbfd5d875e2448"
server_key_version: "encryptionkey@0"
I20260620 14:45:58.720472 2184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 14:45:58.721120 2199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:58.721268 2184 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:45:58.721362 2184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "83c0797ece354bcda365f0fc36cb476e"
format_stamp: "Formatted at 2026-06-20 14:45:58 on dist-test-slave-cx89"
server_key: "64f263641750f829dc6c2f34af1957ee"
server_key_iv: "f96aaa4c58a07cff84fbfd5d875e2448"
server_key_version: "encryptionkey@0"
I20260620 14:45:58.721459 2184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:45:58.746815 2184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:45:58.749223 2184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:45:58.749366 2184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:45:58.749922 2184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:45:58.749966 2184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:58.749990 2184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:45:58.750003 2184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:45:58 dist-test-slave-cx89 krb5kdc[2067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:45:58.757616 2184 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34179
I20260620 14:45:58.757673 2314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34179 every 8 connection(s)
I20260620 14:45:58.758020 2184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:45:58.759974 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:58.750338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52605 (local address 127.0.98.62:33065)
0620 14:45:58.750408 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:58.750410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:58.750650 (+ 240us) server_negotiation.cc:407] Connection header received
0620 14:45:58.751388 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:58.751390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:58.751407 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:58.751513 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:58.752458 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:58.752998 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:58.754020 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:58.754239 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:58.754296 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:58.757222 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:58.757223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:58.757224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:58.757231 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:58.758752 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:58.759122 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:58.759123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:58.759123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:58.759175 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:58.759360 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:58.759361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:58.759361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:58.759511 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:58.759554 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:58.759829 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 14:45:58.759906 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:45:58.760087 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:58.750314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:52605)
0620 14:45:58.750500 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:58.750553 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:58.751238 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:58.751612 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:58.751618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:58.752019 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:58.752346 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:58.752356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:58.753094 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:58.753095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:58.753936 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:58.753939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:58.754332 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:58.754333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:58.754506 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:58.755003 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:58.755013 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:58.757122 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:58.758824 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:58.758827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:58.758829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:58.759055 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:58.759226 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:58.759226 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:58.759227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:58.759316 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:58.759560 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:58.759563 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:58.759690 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 14:45:58.759775 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:45:58.759915 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260620 14:45:58.760944 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 2184
I20260620 14:45:58.761109 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:45:58.761296 392 external_mini_cluster.cc:1428] Setting key 4ed8494e3d7ad203f646051e85337dc4
I20260620 14:45:58.765103 2315 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33065
I20260620 14:45:58.765257 2315 heartbeater.cc:464] Registering TS with master...
I20260620 14:45:58.765599 2315 heartbeater.cc:511] Master 127.0.98.62:33065 requested a full tablet report, sending...
I20260620 14:45:58.766382 2111 ts_manager.cc:194] Registered new tserver with Master: 83c0797ece354bcda365f0fc36cb476e (127.0.98.1:34179)
I20260620 14:45:58.767585 2111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52605
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:45:58.772919 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:58.766882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44379 (local address 127.0.98.62:33065)
0620 14:45:58.766926 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:58.766928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:58.767230 (+ 302us) server_negotiation.cc:407] Connection header received
0620 14:45:58.767284 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:58.767287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:58.767298 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:58.767362 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:58.767759 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:58.768456 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:58.769155 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:58.769453 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:58.769530 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:58.770671 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:58.770672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:58.770672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:58.770679 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:58.771910 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:58.772275 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:58.772276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:58.772277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:58.772329 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:58.772500 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:58.772501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:58.772501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:58.772610 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:58.772656 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:58.772790 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 14:45:58.772854 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:45:58.773108 2322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:58.767010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.1:44379)
0620 14:45:58.767164 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:58.767175 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:58.767234 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:58.767365 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:58.767365 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:58.767597 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:58.767668 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:58.767670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:58.768562 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:58.768563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:58.769075 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:58.769076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:58.769530 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:58.769531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:58.769600 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:58.770140 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:58.770144 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:58.770510 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:58.771990 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:58.771991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:58.771992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:58.772205 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:58.772386 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:58.772386 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:58.772387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:58.772452 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:58.772657 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:58.772658 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:58.772706 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:45:58.772771 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:45:58.772873 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:45:58.786653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33065
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:45:58.864377 2324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:45:58.864593 2324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:45:58.864624 2324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:45:58.865780 2324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:45:58.865821 2324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:45:58.865849 2324 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:45:58.865864 2324 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:45:58.865893 2324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:45:58.867322 2324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:45:58.867583 2324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:45:58.867782 2324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:45:58.872454 2332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:58.872465 2329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:58.872509 2330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:58.872754 2324 server_base.cc:1034] running on GCE node
I20260620 14:45:58.872924 2324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:45:58.873149 2324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:45:58.874286 2324 hybrid_clock.cc:648] HybridClock initialized: now 1781966758874271 us; error 29 us; skew 500 ppm
Jun 20 14:45:58 dist-test-slave-cx89 krb5kdc[2067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:45:58.876724 2324 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:45:58.877128 2324 webserver.cc:460] Webserver started at http://127.0.98.2:39755/ using document root <none> and password file <none>
I20260620 14:45:58.877313 2324 fs_manager.cc:359] Metadata directory not provided
I20260620 14:45:58.877355 2324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:45:58.877473 2324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:45:58.878528 2324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "6a0180a125c241169a838b3aa130326e"
format_stamp: "Formatted at 2026-06-20 14:45:58 on dist-test-slave-cx89"
server_key: "963bde040a3bcdd69bb101564a3cecc6"
server_key_iv: "0603154d9839841c51a57a91d4f19113"
server_key_version: "encryptionkey@0"
I20260620 14:45:58.878841 2324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6a0180a125c241169a838b3aa130326e"
format_stamp: "Formatted at 2026-06-20 14:45:58 on dist-test-slave-cx89"
server_key: "963bde040a3bcdd69bb101564a3cecc6"
server_key_iv: "0603154d9839841c51a57a91d4f19113"
server_key_version: "encryptionkey@0"
I20260620 14:45:58.880100 2324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:45:58.880759 2339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:58.880914 2324 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:45:58.880988 2324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "6a0180a125c241169a838b3aa130326e"
format_stamp: "Formatted at 2026-06-20 14:45:58 on dist-test-slave-cx89"
server_key: "963bde040a3bcdd69bb101564a3cecc6"
server_key_iv: "0603154d9839841c51a57a91d4f19113"
server_key_version: "encryptionkey@0"
I20260620 14:45:58.881050 2324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:45:58.901871 2324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:45:58.903968 2324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:45:58.904084 2324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:45:58.904577 2324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:45:58.904616 2324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:58.904639 2324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:45:58.904652 2324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:45:58 dist-test-slave-cx89 krb5kdc[2067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:45:58.911095 2324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32781
I20260620 14:45:58.911334 2454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32781 every 8 connection(s)
I20260620 14:45:58.911465 2324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:45:58.913406 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:58.905059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44835 (local address 127.0.98.62:33065)
0620 14:45:58.905106 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:58.905108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:58.905329 (+ 221us) server_negotiation.cc:407] Connection header received
0620 14:45:58.905854 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:58.905856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:58.905869 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:58.905950 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:58.906716 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:58.907496 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:58.908490 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:58.908692 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:58.908747 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:58.910805 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:58.910807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:58.910808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:58.910815 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:58.912221 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:58.912576 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:58.912577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:58.912578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:58.912631 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:58.912833 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:58.912834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:58.912834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:58.912944 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:58.912997 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:58.913246 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 14:45:58.913319 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:45:58.913560 2348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:58.905059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:44835)
0620 14:45:58.905217 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:58.905250 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:58.905741 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:58.906027 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:58.906031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:58.906347 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:58.906620 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:58.906625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:58.907584 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:58.907585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:58.908407 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:58.908409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:58.908768 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:58.908770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:58.908885 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:58.909300 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:58.909308 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:58.910716 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:58.912290 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:58.912296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:58.912298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:58.912510 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:58.912682 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:58.912683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:58.912683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:58.912781 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:58.912995 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:58.913001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:58.913133 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 14:45:58.913217 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:45:58.913361 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260620 14:45:58.913992 2455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33065
I20260620 14:45:58.914077 2455 heartbeater.cc:464] Registering TS with master...
I20260620 14:45:58.914255 2455 heartbeater.cc:511] Master 127.0.98.62:33065 requested a full tablet report, sending...
I20260620 14:45:58.914616 2112 ts_manager.cc:194] Registered new tserver with Master: 6a0180a125c241169a838b3aa130326e (127.0.98.2:32781)
I20260620 14:45:58.914987 2112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44835
I20260620 14:45:58.919891 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:58.914976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60937 (local address 127.0.98.62:33065)
0620 14:45:58.915067 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:58.915069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:58.915186 (+ 117us) server_negotiation.cc:407] Connection header received
0620 14:45:58.915306 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:58.915307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:58.915316 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:58.915361 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:58.915679 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:58.916179 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:58.916817 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:58.916980 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:58.917030 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:58.917699 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:58.917700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:58.917700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:58.917705 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:58.918868 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:58.919135 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:58.919137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:58.919137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:58.919194 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:58.919380 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:58.919380 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:58.919380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:58.919564 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:58.919618 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:58.919774 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 14:45:58.919831 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:45:58.919910 2461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:58.914921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.2:60937)
0620 14:45:58.915124 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:58.915135 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:58.915256 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:58.915373 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:58.915374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:58.915560 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:58.915616 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:58.915618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:58.916261 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:58.916261 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:58.916760 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:58.916761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:58.917028 (+ 267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:58.917029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:58.917078 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:58.917392 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:58.917396 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:58.917638 (+ 242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:58.918922 (+ 1284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:58.918923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:58.918923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:58.919062 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:58.919274 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:58.919275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:58.919276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:58.919335 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:58.919641 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:58.919642 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:58.919683 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:45:58.919756 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:45:58.919835 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20260620 14:45:58.921259 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 2324
I20260620 14:45:58.921335 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:45:58.921490 392 external_mini_cluster.cc:1428] Setting key bc11f42e2011e7fcb19b2b7c6016c6ec
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:45:58.943943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33065
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:45:59.023506 2464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:45:59.023705 2464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:45:59.023727 2464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:45:59.024890 2464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:45:59.024930 2464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:45:59.024950 2464 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:45:59.024971 2464 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:45:59.024992 2464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:45:59.026414 2464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:45:59.026636 2464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:45:59.026836 2464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:45:59.031472 2469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:45:59.031432 2470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:59.031572 2464 server_base.cc:1034] running on GCE node
W20260620 14:45:59.031489 2472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:45:59.031787 2464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:45:59.031994 2464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:45:59.033138 2464 hybrid_clock.cc:648] HybridClock initialized: now 1781966759033128 us; error 27 us; skew 500 ppm
Jun 20 14:45:59 dist-test-slave-cx89 krb5kdc[2067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:45:59.035485 2464 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:45:59.035890 2464 webserver.cc:460] Webserver started at http://127.0.98.3:43961/ using document root <none> and password file <none>
I20260620 14:45:59.036083 2464 fs_manager.cc:359] Metadata directory not provided
I20260620 14:45:59.036123 2464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:45:59.036208 2464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:45:59.037170 2464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "19b7d50b7b25407392d1009864c83f23"
format_stamp: "Formatted at 2026-06-20 14:45:59 on dist-test-slave-cx89"
server_key: "2e300a694a95a11dda729204328d9a04"
server_key_iv: "c2e1cecbda3ffa6267919191d4894ca9"
server_key_version: "encryptionkey@0"
I20260620 14:45:59.037442 2464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "19b7d50b7b25407392d1009864c83f23"
format_stamp: "Formatted at 2026-06-20 14:45:59 on dist-test-slave-cx89"
server_key: "2e300a694a95a11dda729204328d9a04"
server_key_iv: "c2e1cecbda3ffa6267919191d4894ca9"
server_key_version: "encryptionkey@0"
I20260620 14:45:59.038537 2464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:45:59.039131 2479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:59.039268 2464 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:45:59.039343 2464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "19b7d50b7b25407392d1009864c83f23"
format_stamp: "Formatted at 2026-06-20 14:45:59 on dist-test-slave-cx89"
server_key: "2e300a694a95a11dda729204328d9a04"
server_key_iv: "c2e1cecbda3ffa6267919191d4894ca9"
server_key_version: "encryptionkey@0"
I20260620 14:45:59.039395 2464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:45:59.051716 2464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:45:59.053290 2464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:45:59.053390 2464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:45:59.053810 2464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:45:59.053843 2464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:59.053862 2464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:45:59.053874 2464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:59.059991 2464 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36833
I20260620 14:45:59.060098 2594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36833 every 8 connection(s)
Jun 20 14:45:59 dist-test-slave-cx89 krb5kdc[2067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:45:59.060333 2464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:45:59.063256 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.054214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48247 (local address 127.0.98.62:33065)
0620 14:45:59.054275 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:59.054278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:59.054458 (+ 180us) server_negotiation.cc:407] Connection header received
0620 14:45:59.054965 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:59.054967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:59.054979 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:59.055040 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:59.055961 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.056711 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.058139 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.058431 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.058488 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.060838 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:59.060839 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:59.060839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:59.060846 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:59.062075 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:59.062422 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:59.062423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:59.062424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:59.062473 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:59.062682 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:59.062684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:59.062684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:59.062785 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:59.062838 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:59.063089 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 14:45:59.063172 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:45:59.063397 2488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.054188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:48247)
0620 14:45:59.054348 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:59.054388 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:59.054877 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:59.055146 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:59.055151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:59.055555 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:59.055869 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.055877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.056795 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.056796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.058059 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.058061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.058508 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.058509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.058655 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.059187 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:59.059197 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:59.060746 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:59.062153 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:59.062159 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:59.062161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:59.062360 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:59.062524 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:59.062524 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:59.062525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:59.062632 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:59.062837 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:59.062843 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:59.062970 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 14:45:59.063054 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:45:59.063195 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260620 14:45:59.063831 2595 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33065
I20260620 14:45:59.063907 2595 heartbeater.cc:464] Registering TS with master...
I20260620 14:45:59.064069 2595 heartbeater.cc:511] Master 127.0.98.62:33065 requested a full tablet report, sending...
I20260620 14:45:59.064402 2111 ts_manager.cc:194] Registered new tserver with Master: 19b7d50b7b25407392d1009864c83f23 (127.0.98.3:36833)
I20260620 14:45:59.064895 2111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48247
I20260620 14:45:59.068070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 2464
I20260620 14:45:59.068169 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:45:59.068342 392 external_mini_cluster.cc:1428] Setting key 041a204360bf8b37f058b82e18a7b02e
I20260620 14:45:59.069103 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:45:59.069618 2601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.064696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.98.3:53009)
0620 14:45:59.064812 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:59.064822 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:59.064878 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:59.065052 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:59.065053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:59.065189 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:59.065236 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.065237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.065951 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.065952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.066436 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.066437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.066700 (+ 263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.066700 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.066747 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.067056 (+ 309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:59.067060 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:59.067290 (+ 230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:59.068705 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:59.068706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:59.068707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:59.068898 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:59.069086 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:59.069087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:59.069087 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:59.069148 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:59.069384 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:59.069384 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:59.069416 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 14:45:59.069471 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:45:59.069551 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
I20260620 14:45:59.069669 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.064741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53009 (local address 127.0.98.62:33065)
0620 14:45:59.064781 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:59.064782 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:59.064865 (+ 83us) server_negotiation.cc:407] Connection header received
0620 14:45:59.064928 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:59.064929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:59.064937 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:59.064988 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:59.065297 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.065882 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.066489 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.066656 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.066695 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.067349 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:59.067350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:59.067350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:59.067355 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:59.068641 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:59.068956 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:59.068957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:59.068957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:59.069036 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:59.069202 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:59.069203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:59.069203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:59.069325 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:59.069383 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:59.069507 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 14:45:59.069588 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Jun 20 14:45:59 dist-test-slave-cx89 krb5kdc[2067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:45:59 dist-test-slave-cx89 krb5kdc[2067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:45:59.089654 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.083372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.0.98.62:33065)
0620 14:45:59.083436 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:59.083438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:59.083447 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:45:59.083515 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:59.083516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:59.083525 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:59.083566 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:59.084134 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.084619 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.085323 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.085508 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.085560 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.087666 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:59.087667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:59.087667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:59.087674 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:59.088678 (+ 1004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:59.088970 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:59.088971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:59.088971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:59.089019 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:59.089174 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:59.089175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:59.089175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:59.089285 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:59.089369 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:59.089488 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 14:45:59.089563 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:45:59.091212 2111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:45:59.092727 2111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:45:59.098469 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.094994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51434 (local address 127.0.98.1:34179)
0620 14:45:59.095243 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:59.095247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:59.095254 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:45:59.095282 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:59.095283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:59.095294 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:59.095348 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:45:59.095501 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.096275 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.097459 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.098116 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.098147 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.098182 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:59.098347 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 14:45:59.098383 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":73,"threads_started":1}
I20260620 14:45:59.098461 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.094831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34179 (local address 127.0.0.1:51434)
0620 14:45:59.095132 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:59.095140 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:59.095172 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:59.095360 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:59.095361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:59.095363 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:45:59.095394 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.095396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.096344 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.096345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.097349 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.097352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.098225 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.098225 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.098266 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.098269 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:45:59.098313 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 14:45:59.098376 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":1,"thread_start_us":43,"threads_started":1}
I20260620 14:45:59.098503 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.094819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35908 (local address 127.0.98.2:32781)
0620 14:45:59.094958 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:59.094963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:59.094972 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:45:59.095006 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:59.095007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:59.095020 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:59.095082 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:45:59.095228 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.096398 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.097465 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.098004 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.098065 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.098119 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:59.098356 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 14:45:59.098391 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260620 14:45:59.098690 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.094651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32781 (local address 127.0.0.1:35908)
0620 14:45:59.094766 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:59.094810 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:59.094875 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:59.095094 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:59.095097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:59.095099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:45:59.095160 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.095161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.096499 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.096500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.097369 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.097372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.098150 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.098151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.098243 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.098256 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:45:59.098316 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:45:59.098627 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":264,"thread_start_us":52,"threads_started":1}
I20260620 14:45:59.098948 2609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.094651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36833 (local address 127.0.0.1:42250)
0620 14:45:59.094983 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:59.094993 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:59.095042 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:59.095668 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:59.095669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:59.095671 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:45:59.095709 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.095711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.096914 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.096915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.097918 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.097921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.098742 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.098743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.098801 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.098804 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:45:59.098848 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 14:45:59.098884 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":4,"spinlock_wait_cycles":896,"thread_start_us":62,"threads_started":1}
I20260620 14:45:59.099001 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.094727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42250 (local address 127.0.98.3:36833)
0620 14:45:59.095477 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:59.095482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:59.095490 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:45:59.095522 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:59.095524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:59.095537 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:59.095610 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:45:59.095771 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.096824 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.097999 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.098659 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.098726 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.098779 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:59.098873 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 14:45:59.098904 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":100,"threads_started":1}
I20260620 14:45:59.099306 2389 tablet_service.cc:1467] Processing CreateTablet for tablet e8388147f3bc41a98e6ade794becebdb (DEFAULT_TABLE table=test-table [id=b08cee5b26b5400cb38f8803c333315f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:45:59.099690 2389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8388147f3bc41a98e6ade794becebdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:59.099648 2249 tablet_service.cc:1467] Processing CreateTablet for tablet e8388147f3bc41a98e6ade794becebdb (DEFAULT_TABLE table=test-table [id=b08cee5b26b5400cb38f8803c333315f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:45:59.099964 2249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8388147f3bc41a98e6ade794becebdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:59.099972 2529 tablet_service.cc:1467] Processing CreateTablet for tablet e8388147f3bc41a98e6ade794becebdb (DEFAULT_TABLE table=test-table [id=b08cee5b26b5400cb38f8803c333315f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:45:59.100281 2529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8388147f3bc41a98e6ade794becebdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:59.102519 2615 tablet_bootstrap.cc:492] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23: Bootstrap starting.
I20260620 14:45:59.103201 2617 tablet_bootstrap.cc:492] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e: Bootstrap starting.
I20260620 14:45:59.103461 2615 tablet_bootstrap.cc:654] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:59.103545 2616 tablet_bootstrap.cc:492] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e: Bootstrap starting.
I20260620 14:45:59.103798 2615 log.cc:824] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23: Log is configured to *not* fsync() on all Append() calls
I20260620 14:45:59.103848 2617 tablet_bootstrap.cc:654] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:59.104106 2616 tablet_bootstrap.cc:654] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:59.104131 2617 log.cc:824] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:45:59.104410 2616 log.cc:824] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:45:59.104530 2615 tablet_bootstrap.cc:492] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23: No bootstrap required, opened a new log
I20260620 14:45:59.104596 2615 ts_tablet_manager.cc:1397] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:45:59.104743 2617 tablet_bootstrap.cc:492] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e: No bootstrap required, opened a new log
I20260620 14:45:59.104799 2617 ts_tablet_manager.cc:1397] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:45:59.105965 2615 raft_consensus.cc:348] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.106125 2615 raft_consensus.cc:374] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:59.106165 2615 raft_consensus.cc:729] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b7d50b7b25407392d1009864c83f23, State: Initialized, Role: FOLLOWER
I20260620 14:45:59.106086 2617 raft_consensus.cc:348] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.106230 2617 raft_consensus.cc:374] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:59.106262 2617 raft_consensus.cc:729] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c0797ece354bcda365f0fc36cb476e, State: Initialized, Role: FOLLOWER
I20260620 14:45:59.106272 2615 consensus_queue.cc:260] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.106362 2617 consensus_queue.cc:260] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.106541 2615 ts_tablet_manager.cc:1428] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:45:59.106552 2595 heartbeater.cc:503] Master 127.0.98.62:33065 was elected leader, sending a full tablet report...
I20260620 14:45:59.106601 2617 ts_tablet_manager.cc:1428] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:45:59.106702 2315 heartbeater.cc:503] Master 127.0.98.62:33065 was elected leader, sending a full tablet report...
I20260620 14:45:59.107235 2616 tablet_bootstrap.cc:492] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e: No bootstrap required, opened a new log
I20260620 14:45:59.107303 2616 ts_tablet_manager.cc:1397] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 14:45:59.108719 2616 raft_consensus.cc:348] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.108835 2616 raft_consensus.cc:374] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:59.108855 2616 raft_consensus.cc:729] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0180a125c241169a838b3aa130326e, State: Initialized, Role: FOLLOWER
I20260620 14:45:59.108955 2616 consensus_queue.cc:260] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.109171 2616 ts_tablet_manager.cc:1428] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:45:59.109218 2455 heartbeater.cc:503] Master 127.0.98.62:33065 was elected leader, sending a full tablet report...
I20260620 14:45:59.157321 2623 raft_consensus.cc:482] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:45:59.157480 2623 raft_consensus.cc:504] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.157917 2623 leader_election.cc:288] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c0797ece354bcda365f0fc36cb476e (127.0.98.1:34179), 19b7d50b7b25407392d1009864c83f23 (127.0.98.3:36833)
I20260620 14:45:59.160820 2348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.158076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36833 (local address 127.0.98.2:54659)
0620 14:45:59.158130 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:59.158143 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:59.158201 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:59.158499 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:59.158500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:59.158503 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:45:59.158575 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.158577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.159219 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.159220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.160067 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.160069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.160577 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.160578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.160653 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.160656 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:45:59.160700 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 14:45:59.160739 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":53}
I20260620 14:45:59.160808 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.158150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54171 (local address 127.0.98.1:34179)
0620 14:45:59.158208 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:59.158210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:59.158463 (+ 253us) server_negotiation.cc:407] Connection header received
0620 14:45:59.158521 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:59.158522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:59.158532 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:59.158596 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:45:59.158724 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.159159 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.160147 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.160511 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.160569 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.160578 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:59.160715 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 14:45:59.160747 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:45:59.160813 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.158301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54659 (local address 127.0.98.3:36833)
0620 14:45:59.158360 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:59.158361 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:59.158372 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:45:59.158408 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:59.158409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:59.158422 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:59.158485 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:45:59.158640 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.159144 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.160149 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.160505 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.160573 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.160583 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:59.160713 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 14:45:59.160745 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:45:59.160844 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.158236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34179 (local address 127.0.98.2:54171)
0620 14:45:59.158401 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:59.158410 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:59.158468 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:59.158599 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:59.158600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:59.158601 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:45:59.158671 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.158672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.159230 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.159231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.160061 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.160064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.160581 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.160582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.160649 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.160652 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:45:59.160706 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:45:59.160745 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":3,"thread_start_us":104,"threads_started":1}
I20260620 14:45:59.161137 2549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8388147f3bc41a98e6ade794becebdb" candidate_uuid: "6a0180a125c241169a838b3aa130326e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b7d50b7b25407392d1009864c83f23" is_pre_election: true
I20260620 14:45:59.161164 2269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8388147f3bc41a98e6ade794becebdb" candidate_uuid: "6a0180a125c241169a838b3aa130326e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c0797ece354bcda365f0fc36cb476e" is_pre_election: true
I20260620 14:45:59.161278 2549 raft_consensus.cc:2454] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0180a125c241169a838b3aa130326e in term 0.
I20260620 14:45:59.161289 2269 raft_consensus.cc:2454] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0180a125c241169a838b3aa130326e in term 0.
I20260620 14:45:59.161495 2342 leader_election.cc:302] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0180a125c241169a838b3aa130326e, 83c0797ece354bcda365f0fc36cb476e; no voters:
I20260620 14:45:59.161623 2623 raft_consensus.cc:2781] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:45:59.161684 2623 raft_consensus.cc:482] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:45:59.161710 2623 raft_consensus.cc:3037] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Advancing to term 1
W20260620 14:45:59.161955 2456 tablet.cc:2369] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:45:59.162684 2623 raft_consensus.cc:504] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.162820 2623 leader_election.cc:288] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [CANDIDATE]: Term 1 election: Requested vote from peers 83c0797ece354bcda365f0fc36cb476e (127.0.98.1:34179), 19b7d50b7b25407392d1009864c83f23 (127.0.98.3:36833)
I20260620 14:45:59.162976 2549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8388147f3bc41a98e6ade794becebdb" candidate_uuid: "6a0180a125c241169a838b3aa130326e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b7d50b7b25407392d1009864c83f23"
I20260620 14:45:59.162989 2269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8388147f3bc41a98e6ade794becebdb" candidate_uuid: "6a0180a125c241169a838b3aa130326e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c0797ece354bcda365f0fc36cb476e"
I20260620 14:45:59.163054 2269 raft_consensus.cc:3037] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:59.163059 2549 raft_consensus.cc:3037] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:59.163789 2269 raft_consensus.cc:2454] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0180a125c241169a838b3aa130326e in term 1.
I20260620 14:45:59.163865 2549 raft_consensus.cc:2454] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0180a125c241169a838b3aa130326e in term 1.
I20260620 14:45:59.163940 2342 leader_election.cc:302] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0180a125c241169a838b3aa130326e, 83c0797ece354bcda365f0fc36cb476e; no voters:
I20260620 14:45:59.164034 2623 raft_consensus.cc:2781] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:45:59.164186 2623 raft_consensus.cc:686] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [term 1 LEADER]: Becoming Leader. State: Replica: 6a0180a125c241169a838b3aa130326e, State: Running, Role: LEADER
I20260620 14:45:59.164266 2623 consensus_queue.cc:237] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.165099 2111 catalog_manager.cc:5495] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0180a125c241169a838b3aa130326e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6a0180a125c241169a838b3aa130326e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:45:59 dist-test-slave-cx89 krb5kdc[2067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:45:59.198215 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.192085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33065 (local address 127.0.0.1:51774)
0620 14:45:59.192254 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:59.192265 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:59.192327 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:59.192439 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:59.192439 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:59.192740 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:45:59.192784 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.192786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.193435 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.193436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.194034 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.194036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.194418 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.194419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.194466 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.194952 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:45:59.194956 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:45:59.196232 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:45:59.197426 (+ 1194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:59.197432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:59.197439 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:59.197615 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:59.197747 (+ 132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:45:59.197747 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:45:59.197747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:45:59.197792 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:45:59.197950 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:45:59.197954 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:45:59.197989 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:45:59.198044 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:45:59.198123 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260620 14:45:59.198259 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.192163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51774 (local address 127.0.98.62:33065)
0620 14:45:59.192231 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:59.192233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:59.192323 (+ 90us) server_negotiation.cc:407] Connection header received
0620 14:45:59.192368 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:59.192368 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:59.192377 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:59.192422 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:45:59.192872 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.193354 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.194126 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.194315 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.194372 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.196303 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:45:59.196304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:45:59.196304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:45:59.196311 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:45:59.197355 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:59.197666 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:59.197667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:59.197667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:59.197706 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:45:59.197829 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:45:59.197829 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:45:59.197830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:45:59.197912 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:45:59.197953 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:59.198064 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 14:45:59.198163 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":41}
I20260620 14:45:59.200084 2111 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51774:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:45:59.200270 2111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:45:59.201627 2249 tablet_service.cc:1467] Processing CreateTablet for tablet 83ba0eed69db4c2db1fdc5cf4530b9a0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b15bcd0cda08462faeaa2f41229a051e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:45:59.201647 2529 tablet_service.cc:1467] Processing CreateTablet for tablet 83ba0eed69db4c2db1fdc5cf4530b9a0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b15bcd0cda08462faeaa2f41229a051e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:45:59.201773 2529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ba0eed69db4c2db1fdc5cf4530b9a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:59.201773 2249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ba0eed69db4c2db1fdc5cf4530b9a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:59.202827 2615 tablet_bootstrap.cc:492] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23: Bootstrap starting.
I20260620 14:45:59.202924 2389 tablet_service.cc:1467] Processing CreateTablet for tablet 83ba0eed69db4c2db1fdc5cf4530b9a0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b15bcd0cda08462faeaa2f41229a051e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:45:59.203071 2389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ba0eed69db4c2db1fdc5cf4530b9a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:59.203274 2615 tablet_bootstrap.cc:654] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:59.203593 2617 tablet_bootstrap.cc:492] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e: Bootstrap starting.
I20260620 14:45:59.203888 2615 tablet_bootstrap.cc:492] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23: No bootstrap required, opened a new log
I20260620 14:45:59.203928 2615 ts_tablet_manager.cc:1397] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:45:59.204092 2617 tablet_bootstrap.cc:654] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:59.204067 2615 raft_consensus.cc:348] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.204133 2615 raft_consensus.cc:374] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:59.204150 2615 raft_consensus.cc:729] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b7d50b7b25407392d1009864c83f23, State: Initialized, Role: FOLLOWER
I20260620 14:45:59.204190 2615 consensus_queue.cc:260] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.204207 2616 tablet_bootstrap.cc:492] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 6a0180a125c241169a838b3aa130326e: Bootstrap starting.
I20260620 14:45:59.204267 2615 ts_tablet_manager.cc:1428] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 14:45:59.204401 2621 tablet_replica.cc:439] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } } }
I20260620 14:45:59.204489 2621 tablet_replica.cc:442] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:45:59.204589 2616 tablet_bootstrap.cc:654] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 6a0180a125c241169a838b3aa130326e: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:59.204777 2617 tablet_bootstrap.cc:492] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e: No bootstrap required, opened a new log
I20260620 14:45:59.204825 2617 ts_tablet_manager.cc:1397] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:45:59.204965 2617 raft_consensus.cc:348] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.205096 2617 raft_consensus.cc:374] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:59.205138 2617 raft_consensus.cc:729] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c0797ece354bcda365f0fc36cb476e, State: Initialized, Role: FOLLOWER
I20260620 14:45:59.205190 2617 consensus_queue.cc:260] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.205222 2616 tablet_bootstrap.cc:492] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 6a0180a125c241169a838b3aa130326e: No bootstrap required, opened a new log
I20260620 14:45:59.205265 2616 ts_tablet_manager.cc:1397] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 6a0180a125c241169a838b3aa130326e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:45:59.205300 2617 ts_tablet_manager.cc:1428] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:59.205379 2622 tablet_replica.cc:439] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } } }
I20260620 14:45:59.205410 2616 raft_consensus.cc:348] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.205456 2622 tablet_replica.cc:442] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:45:59.205467 2616 raft_consensus.cc:374] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:59.205492 2616 raft_consensus.cc:729] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0180a125c241169a838b3aa130326e, State: Initialized, Role: FOLLOWER
I20260620 14:45:59.205538 2616 consensus_queue.cc:260] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 6a0180a125c241169a838b3aa130326e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.205634 2616 ts_tablet_manager.cc:1428] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 6a0180a125c241169a838b3aa130326e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:45:59.205665 2623 tablet_replica.cc:439] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 6a0180a125c241169a838b3aa130326e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } } }
I20260620 14:45:59.205741 2623 tablet_replica.cc:442] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 6a0180a125c241169a838b3aa130326e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 14:45:59.258724 2316 tablet.cc:2369] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:45:59.310935 2596 tablet.cc:2369] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:45:59.355903 2622 raft_consensus.cc:482] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:45:59.356005 2622 raft_consensus.cc:504] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.356339 2622 leader_election.cc:288] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0180a125c241169a838b3aa130326e (127.0.98.2:32781), 19b7d50b7b25407392d1009864c83f23 (127.0.98.3:36833)
I20260620 14:45:59.358959 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.356574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44353 (local address 127.0.98.3:36833)
0620 14:45:59.356618 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:59.356620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:59.356714 (+ 94us) server_negotiation.cc:407] Connection header received
0620 14:45:59.356780 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:59.356781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:59.356789 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:59.356831 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:45:59.356971 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.357427 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.358327 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.358664 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.358721 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.358731 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:59.358859 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 14:45:59.358891 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:45:59.358989 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.356511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36833 (local address 127.0.98.1:44353)
0620 14:45:59.356643 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:59.356652 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:59.356713 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:59.356846 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:59.356846 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:59.356848 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:45:59.356916 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.356917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.357498 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.357499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.358244 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.358246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.358732 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.358733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.358800 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.358803 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:45:59.358846 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 14:45:59.358887 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":21,"thread_start_us":52,"threads_started":1}
I20260620 14:45:59.359035 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.356605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54077 (local address 127.0.98.2:32781)
0620 14:45:59.356645 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:59.356647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:59.356706 (+ 59us) server_negotiation.cc:407] Connection header received
0620 14:45:59.356770 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:59.356771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:59.356782 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:59.356826 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:45:59.356961 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.357490 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.358371 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.358735 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.358794 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.358805 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:59.358936 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 14:45:59.358977 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:45:59.359239 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.356511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32781 (local address 127.0.98.1:54077)
0620 14:45:59.356642 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:59.356652 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:59.356713 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:59.356841 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:59.356842 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:59.356844 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:45:59.356911 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.356912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.357587 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.357588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.358293 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.358295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.358813 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.358814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.358878 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.358882 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:45:59.358925 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 14:45:59.358967 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260620 14:45:59.359207 2549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ba0eed69db4c2db1fdc5cf4530b9a0" candidate_uuid: "83c0797ece354bcda365f0fc36cb476e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b7d50b7b25407392d1009864c83f23" is_pre_election: true
I20260620 14:45:59.359288 2549 raft_consensus.cc:2454] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c0797ece354bcda365f0fc36cb476e in term 0.
I20260620 14:45:59.359485 2203 leader_election.cc:302] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b7d50b7b25407392d1009864c83f23, 83c0797ece354bcda365f0fc36cb476e; no voters:
I20260620 14:45:59.359584 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ba0eed69db4c2db1fdc5cf4530b9a0" candidate_uuid: "83c0797ece354bcda365f0fc36cb476e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0180a125c241169a838b3aa130326e" is_pre_election: true
I20260620 14:45:59.359637 2622 raft_consensus.cc:2781] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:45:59.359674 2622 raft_consensus.cc:482] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:45:59.359700 2622 raft_consensus.cc:3037] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:59.359691 2409 raft_consensus.cc:2454] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c0797ece354bcda365f0fc36cb476e in term 0.
I20260620 14:45:59.360366 2622 raft_consensus.cc:504] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.360494 2622 leader_election.cc:288] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e [CANDIDATE]: Term 1 election: Requested vote from peers 6a0180a125c241169a838b3aa130326e (127.0.98.2:32781), 19b7d50b7b25407392d1009864c83f23 (127.0.98.3:36833)
I20260620 14:45:59.360666 2549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ba0eed69db4c2db1fdc5cf4530b9a0" candidate_uuid: "83c0797ece354bcda365f0fc36cb476e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b7d50b7b25407392d1009864c83f23"
I20260620 14:45:59.360745 2549 raft_consensus.cc:3037] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:59.361265 2549 raft_consensus.cc:2454] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c0797ece354bcda365f0fc36cb476e in term 1.
I20260620 14:45:59.361358 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ba0eed69db4c2db1fdc5cf4530b9a0" candidate_uuid: "83c0797ece354bcda365f0fc36cb476e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0180a125c241169a838b3aa130326e"
I20260620 14:45:59.361397 2203 leader_election.cc:302] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b7d50b7b25407392d1009864c83f23, 83c0797ece354bcda365f0fc36cb476e; no voters:
I20260620 14:45:59.361433 2409 raft_consensus.cc:3037] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:59.361480 2622 raft_consensus.cc:2781] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:45:59.361601 2622 raft_consensus.cc:686] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e [term 1 LEADER]: Becoming Leader. State: Replica: 83c0797ece354bcda365f0fc36cb476e, State: Running, Role: LEADER
I20260620 14:45:59.361699 2622 consensus_queue.cc:237] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.362200 2409 raft_consensus.cc:2454] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 6a0180a125c241169a838b3aa130326e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c0797ece354bcda365f0fc36cb476e in term 1.
I20260620 14:45:59.362140 2638 tablet_replica.cc:439] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e: TxnStatusTablet state changed. Reason: New leader 83c0797ece354bcda365f0fc36cb476e. Latest consensus state: current_term: 1 leader_uuid: "83c0797ece354bcda365f0fc36cb476e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } } }
I20260620 14:45:59.362260 2638 tablet_replica.cc:442] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:45:59.362448 2640 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:45:59.362556 2640 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:45:59.362540 2112 catalog_manager.cc:5495] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c0797ece354bcda365f0fc36cb476e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "83c0797ece354bcda365f0fc36cb476e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } health_report { overall_health: UNKNOWN } } }
I20260620 14:45:59.387214 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.384901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34179 (local address 127.0.0.1:51438)
0620 14:45:59.384964 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:59.384975 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:59.385032 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:59.385191 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:59.385193 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:59.385194 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:45:59.385243 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.385245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.385842 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.385843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.386418 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.386419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.386744 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.386746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.386812 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.386825 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:45:59.387053 (+ 228us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:45:59.387061 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:45:59.387100 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 14:45:59.387149 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 14:45:59.387208 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.384974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.0.98.1:34179)
0620 14:45:59.385023 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:59.385025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:59.385034 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:45:59.385112 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:59.385112 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:59.385131 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:59.385181 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:45:59.385299 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.385774 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.386494 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.386690 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.386741 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.386900 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:45:59.387002 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:45:59.387068 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:59.387112 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 14:45:59.387132 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:45:59.389072 2549 raft_consensus.cc:1261] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23 [term 1 FOLLOWER]: Refusing update from remote peer 83c0797ece354bcda365f0fc36cb476e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:45:59.389086 2409 raft_consensus.cc:1261] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 6a0180a125c241169a838b3aa130326e [term 1 FOLLOWER]: Refusing update from remote peer 83c0797ece354bcda365f0fc36cb476e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:45:59.389345 2622 consensus_queue.cc:1035] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e [LEADER]: Connected to new peer: Peer: permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:45:59.389546 2622 consensus_queue.cc:1035] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:45:59.390591 2621 tablet_replica.cc:439] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23: TxnStatusTablet state changed. Reason: New leader 83c0797ece354bcda365f0fc36cb476e. Latest consensus state: current_term: 1 leader_uuid: "83c0797ece354bcda365f0fc36cb476e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } } }
I20260620 14:45:59.390681 2621 tablet_replica.cc:442] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:45:59.391498 2623 tablet_replica.cc:439] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 6a0180a125c241169a838b3aa130326e: TxnStatusTablet state changed. Reason: New leader 83c0797ece354bcda365f0fc36cb476e. Latest consensus state: current_term: 1 leader_uuid: "83c0797ece354bcda365f0fc36cb476e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } } }
I20260620 14:45:59.391587 2623 tablet_replica.cc:442] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 6a0180a125c241169a838b3aa130326e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:45:59.391716 2638 tablet_replica.cc:439] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83c0797ece354bcda365f0fc36cb476e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } } }
I20260620 14:45:59.391801 2638 tablet_replica.cc:442] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:45:59.392577 2638 tablet_replica.cc:439] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83c0797ece354bcda365f0fc36cb476e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } } }
I20260620 14:45:59.392652 2638 tablet_replica.cc:442] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 83c0797ece354bcda365f0fc36cb476e: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:45:59.392851 2644 mvcc.cc:205] Tried to move back new op lower bound from 7298935846455259136 to 7298935846346014720. Current Snapshot: MvccSnapshot[applied={T|T < 7298935846455259136}]
I20260620 14:45:59.392927 2621 tablet_replica.cc:439] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83c0797ece354bcda365f0fc36cb476e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } } }
I20260620 14:45:59.393002 2621 tablet_replica.cc:442] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 19b7d50b7b25407392d1009864c83f23: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:45:59.393337 2623 tablet_replica.cc:439] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 6a0180a125c241169a838b3aa130326e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83c0797ece354bcda365f0fc36cb476e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } } }
I20260620 14:45:59.393415 2623 tablet_replica.cc:442] T 83ba0eed69db4c2db1fdc5cf4530b9a0 P 6a0180a125c241169a838b3aa130326e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:45:59.393793 2646 mvcc.cc:205] Tried to move back new op lower bound from 7298935846455259136 to 7298935846346014720. Current Snapshot: MvccSnapshot[applied={T|T < 7298935846455259136}]
I20260620 14:45:59.396670 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.394203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.0.98.2:32781)
0620 14:45:59.394236 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:59.394237 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:59.394246 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:45:59.394276 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:59.394277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:59.394294 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:59.394350 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:45:59.394518 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.394969 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.396038 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.396217 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.396262 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.396411 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:45:59.396497 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:45:59.396547 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:59.396605 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 14:45:59.396622 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:45:59.401037 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.398854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33265 (local address 127.0.98.1:34179)
0620 14:45:59.398901 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:59.398903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:59.398911 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:45:59.398932 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:59.398933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:59.398942 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:59.399001 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:45:59.399206 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.399707 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.400424 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.400610 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.400661 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.400831 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:45:59.400859 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:45:59.400900 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:59.400949 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 14:45:59.400969 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:45:59.401055 2461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.398777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34179 (local address 127.0.98.2:33265)
0620 14:45:59.398835 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:59.398845 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:59.398882 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:59.399003 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:59.399003 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:59.399005 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:45:59.399121 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.399125 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.399774 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.399775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.400346 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.400348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.400666 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.400667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.400731 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.400745 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:45:59.400905 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:45:59.400907 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:45:59.400955 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:45:59.400996 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 14:45:59.403085 2549 raft_consensus.cc:1261] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23 [term 1 FOLLOWER]: Refusing update from remote peer 6a0180a125c241169a838b3aa130326e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:45:59.403199 2269 raft_consensus.cc:1261] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e [term 1 FOLLOWER]: Refusing update from remote peer 6a0180a125c241169a838b3aa130326e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:45:59.403479 2625 consensus_queue.cc:1035] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [LEADER]: Connected to new peer: Peer: permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:45:59.403625 2625 consensus_queue.cc:1035] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:45:59.406714 2643 mvcc.cc:205] Tried to move back new op lower bound from 7298935846513627136 to 7298935845537427456. Current Snapshot: MvccSnapshot[applied={T|T < 7298935846513627136}]
I20260620 14:45:59.411336 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.408955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32781 (local address 127.0.98.1:45547)
0620 14:45:59.409081 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:45:59.409092 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:45:59.409142 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:45:59.409291 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:45:59.409291 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:45:59.409304 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:45:59.409357 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.409359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.409931 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.409932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.410526 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:45:59.410528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.410914 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.410915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:45:59.410981 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.410997 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:45:59.411185 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:45:59.411186 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:45:59.411227 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 14:45:59.411266 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":72,"threads_started":1}
I20260620 14:45:59.411362 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:45:59.409111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45547 (local address 127.0.98.2:32781)
0620 14:45:59.409143 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 14:45:59.409144 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:45:59.409152 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:45:59.409192 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:45:59.409193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:45:59.409203 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:45:59.409231 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:45:59.409420 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.409865 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.410620 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:45:59.410857 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:45:59.410908 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:45:59.411088 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:45:59.411120 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:45:59.411145 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0620 14:45:59.411266 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 14:45:59.411293 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:45:59.417832 2111 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:45:59.417936 2111 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:45:59.418663 2111 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277: Sending DeleteTablet for 3 replicas of tablet e8388147f3bc41a98e6ade794becebdb
I20260620 14:45:59.418951 2529 tablet_service.cc:1514] Processing DeleteTablet for tablet e8388147f3bc41a98e6ade794becebdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:45:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42250
I20260620 14:45:59.419018 2249 tablet_service.cc:1514] Processing DeleteTablet for tablet e8388147f3bc41a98e6ade794becebdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:45:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51434
I20260620 14:45:59.419137 2389 tablet_service.cc:1514] Processing DeleteTablet for tablet e8388147f3bc41a98e6ade794becebdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:45:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35908
I20260620 14:45:59.419269 2663 tablet_replica.cc:330] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e: stopping tablet replica
I20260620 14:45:59.419400 2663 raft_consensus.cc:2229] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:45:59.419521 2663 raft_consensus.cc:2258] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:45:59.419559 2111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:45:59.419692 2111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:45:59.420785 2529 tablet_service.cc:1467] Processing CreateTablet for tablet 5be4c626da9642d5b092eaa6944fa892 (DEFAULT_TABLE table=test-table [id=2f0792f2c2d745ca8cd808154493fd92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:45:59.420790 2249 tablet_service.cc:1467] Processing CreateTablet for tablet 5be4c626da9642d5b092eaa6944fa892 (DEFAULT_TABLE table=test-table [id=2f0792f2c2d745ca8cd808154493fd92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:45:59.420910 2249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be4c626da9642d5b092eaa6944fa892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:59.420912 2529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be4c626da9642d5b092eaa6944fa892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:59.421936 2617 tablet_bootstrap.cc:492] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e: Bootstrap starting.
I20260620 14:45:59.422173 2615 tablet_bootstrap.cc:492] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23: Bootstrap starting.
I20260620 14:45:59.422310 2617 tablet_bootstrap.cc:654] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:59.422295 2389 tablet_service.cc:1467] Processing CreateTablet for tablet 5be4c626da9642d5b092eaa6944fa892 (DEFAULT_TABLE table=test-table [id=2f0792f2c2d745ca8cd808154493fd92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:45:59.422389 2389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be4c626da9642d5b092eaa6944fa892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:45:59.422513 2615 tablet_bootstrap.cc:654] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:59.423540 2616 tablet_bootstrap.cc:492] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e: Bootstrap starting.
I20260620 14:45:59.423942 2616 tablet_bootstrap.cc:654] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e: Neither blocks nor log segments found. Creating new log.
I20260620 14:45:59.423993 2663 ts_tablet_manager.cc:1905] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:45:59.425659 2617 tablet_bootstrap.cc:492] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e: No bootstrap required, opened a new log
I20260620 14:45:59.425704 2617 ts_tablet_manager.cc:1397] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 14:45:59.425825 2617 raft_consensus.cc:348] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.425891 2617 raft_consensus.cc:374] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:59.425907 2617 raft_consensus.cc:729] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c0797ece354bcda365f0fc36cb476e, State: Initialized, Role: FOLLOWER
I20260620 14:45:59.425943 2617 consensus_queue.cc:260] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.426023 2617 ts_tablet_manager.cc:1428] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:59.426268 2663 ts_tablet_manager.cc:1918] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:45:59.426323 2663 log.cc:1196] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/wal/wals/e8388147f3bc41a98e6ade794becebdb
I20260620 14:45:59.426651 2663 ts_tablet_manager.cc:1939] T e8388147f3bc41a98e6ade794becebdb P 83c0797ece354bcda365f0fc36cb476e: Deleting consensus metadata
I20260620 14:45:59.427014 2664 tablet_replica.cc:330] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e: stopping tablet replica
I20260620 14:45:59.427040 2098 catalog_manager.cc:4841] TS 83c0797ece354bcda365f0fc36cb476e (127.0.98.1:34179): tablet e8388147f3bc41a98e6ade794becebdb (table test-table [id=b08cee5b26b5400cb38f8803c333315f]) successfully deleted
I20260620 14:45:59.427119 2664 raft_consensus.cc:2229] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:45:59.427207 2664 raft_consensus.cc:2258] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:45:59.427493 2662 tablet_replica.cc:330] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23: stopping tablet replica
I20260620 14:45:59.427601 2662 raft_consensus.cc:2229] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:45:59.427656 2615 tablet_bootstrap.cc:492] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23: No bootstrap required, opened a new log
I20260620 14:45:59.427663 2662 raft_consensus.cc:2258] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:45:59.427716 2615 ts_tablet_manager.cc:1397] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260620 14:45:59.427868 2615 raft_consensus.cc:348] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.427932 2615 raft_consensus.cc:374] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:59.427961 2615 raft_consensus.cc:729] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b7d50b7b25407392d1009864c83f23, State: Initialized, Role: FOLLOWER
I20260620 14:45:59.428006 2615 consensus_queue.cc:260] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.428053 2662 ts_tablet_manager.cc:1905] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:45:59.428088 2615 ts_tablet_manager.cc:1428] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:45:59.428030 2664 ts_tablet_manager.cc:1905] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:45:59.429484 2662 ts_tablet_manager.cc:1918] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:45:59.429555 2662 log.cc:1196] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/e8388147f3bc41a98e6ade794becebdb
I20260620 14:45:59.429790 2662 ts_tablet_manager.cc:1939] T e8388147f3bc41a98e6ade794becebdb P 19b7d50b7b25407392d1009864c83f23: Deleting consensus metadata
I20260620 14:45:59.430086 2664 ts_tablet_manager.cc:1918] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:45:59.430145 2664 log.cc:1196] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/e8388147f3bc41a98e6ade794becebdb
I20260620 14:45:59.430215 2099 catalog_manager.cc:4841] TS 19b7d50b7b25407392d1009864c83f23 (127.0.98.3:36833): tablet e8388147f3bc41a98e6ade794becebdb (table test-table [id=b08cee5b26b5400cb38f8803c333315f]) successfully deleted
I20260620 14:45:59.430353 2664 ts_tablet_manager.cc:1939] T e8388147f3bc41a98e6ade794becebdb P 6a0180a125c241169a838b3aa130326e: Deleting consensus metadata
I20260620 14:45:59.430840 2098 catalog_manager.cc:4841] TS 6a0180a125c241169a838b3aa130326e (127.0.98.2:32781): tablet e8388147f3bc41a98e6ade794becebdb (table test-table [id=b08cee5b26b5400cb38f8803c333315f]) successfully deleted
I20260620 14:45:59.435041 2616 tablet_bootstrap.cc:492] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e: No bootstrap required, opened a new log
I20260620 14:45:59.435102 2616 ts_tablet_manager.cc:1397] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260620 14:45:59.435243 2616 raft_consensus.cc:348] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.435315 2616 raft_consensus.cc:374] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:45:59.435335 2616 raft_consensus.cc:729] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0180a125c241169a838b3aa130326e, State: Initialized, Role: FOLLOWER
I20260620 14:45:59.435381 2616 consensus_queue.cc:260] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.435513 2616 ts_tablet_manager.cc:1428] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:45:59.700029 2623 raft_consensus.cc:482] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:45:59.700136 2623 raft_consensus.cc:504] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.700302 2623 leader_election.cc:288] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c0797ece354bcda365f0fc36cb476e (127.0.98.1:34179), 19b7d50b7b25407392d1009864c83f23 (127.0.98.3:36833)
I20260620 14:45:59.700536 2269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be4c626da9642d5b092eaa6944fa892" candidate_uuid: "6a0180a125c241169a838b3aa130326e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c0797ece354bcda365f0fc36cb476e" is_pre_election: true
I20260620 14:45:59.700546 2549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be4c626da9642d5b092eaa6944fa892" candidate_uuid: "6a0180a125c241169a838b3aa130326e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b7d50b7b25407392d1009864c83f23" is_pre_election: true
I20260620 14:45:59.700644 2549 raft_consensus.cc:2454] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0180a125c241169a838b3aa130326e in term 0.
I20260620 14:45:59.700644 2269 raft_consensus.cc:2454] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0180a125c241169a838b3aa130326e in term 0.
I20260620 14:45:59.700793 2342 leader_election.cc:302] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0180a125c241169a838b3aa130326e, 83c0797ece354bcda365f0fc36cb476e; no voters:
I20260620 14:45:59.700889 2623 raft_consensus.cc:2781] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:45:59.700942 2623 raft_consensus.cc:482] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:45:59.700968 2623 raft_consensus.cc:3037] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:59.701603 2623 raft_consensus.cc:504] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.701721 2623 leader_election.cc:288] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [CANDIDATE]: Term 1 election: Requested vote from peers 83c0797ece354bcda365f0fc36cb476e (127.0.98.1:34179), 19b7d50b7b25407392d1009864c83f23 (127.0.98.3:36833)
I20260620 14:45:59.701859 2269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be4c626da9642d5b092eaa6944fa892" candidate_uuid: "6a0180a125c241169a838b3aa130326e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c0797ece354bcda365f0fc36cb476e"
I20260620 14:45:59.701864 2549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be4c626da9642d5b092eaa6944fa892" candidate_uuid: "6a0180a125c241169a838b3aa130326e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b7d50b7b25407392d1009864c83f23"
I20260620 14:45:59.701922 2549 raft_consensus.cc:3037] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:59.701936 2269 raft_consensus.cc:3037] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:45:59.702464 2549 raft_consensus.cc:2454] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0180a125c241169a838b3aa130326e in term 1.
I20260620 14:45:59.702486 2269 raft_consensus.cc:2454] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0180a125c241169a838b3aa130326e in term 1.
I20260620 14:45:59.702631 2343 leader_election.cc:302] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b7d50b7b25407392d1009864c83f23, 6a0180a125c241169a838b3aa130326e; no voters:
I20260620 14:45:59.702719 2623 raft_consensus.cc:2781] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:45:59.702760 2623 raft_consensus.cc:686] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [term 1 LEADER]: Becoming Leader. State: Replica: 6a0180a125c241169a838b3aa130326e, State: Running, Role: LEADER
I20260620 14:45:59.702802 2623 consensus_queue.cc:237] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:45:59.703337 2111 catalog_manager.cc:5495] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0180a125c241169a838b3aa130326e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6a0180a125c241169a838b3aa130326e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } health_report { overall_health: UNKNOWN } } }
I20260620 14:45:59.708252 2549 raft_consensus.cc:1261] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23 [term 1 FOLLOWER]: Refusing update from remote peer 6a0180a125c241169a838b3aa130326e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:45:59.708285 2269 raft_consensus.cc:1261] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e [term 1 FOLLOWER]: Refusing update from remote peer 6a0180a125c241169a838b3aa130326e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:45:59.708449 2625 consensus_queue.cc:1035] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:45:59.708559 2623 consensus_queue.cc:1035] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [LEADER]: Connected to new peer: Peer: permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:45:59.712172 2646 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:46:01.228135 2111 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:01.228232 2111 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:01.228981 2111 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277: Sending DeleteTablet for 3 replicas of tablet 5be4c626da9642d5b092eaa6944fa892
I20260620 14:46:01.229225 2249 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be4c626da9642d5b092eaa6944fa892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51434
I20260620 14:46:01.229249 2529 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be4c626da9642d5b092eaa6944fa892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42250
I20260620 14:46:01.229251 2389 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be4c626da9642d5b092eaa6944fa892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35908
I20260620 14:46:01.229437 2696 tablet_replica.cc:330] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e: stopping tablet replica
I20260620 14:46:01.229504 2696 raft_consensus.cc:2229] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:46:01.229564 2696 raft_consensus.cc:2258] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:01.229666 2694 tablet_replica.cc:330] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e: stopping tablet replica
I20260620 14:46:01.229718 2694 raft_consensus.cc:2229] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:46:01.229753 2694 raft_consensus.cc:2258] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:01.229933 2696 ts_tablet_manager.cc:1905] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:01.231596 2695 tablet_replica.cc:330] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23: stopping tablet replica
I20260620 14:46:01.231801 2694 ts_tablet_manager.cc:1905] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:01.233287 2694 ts_tablet_manager.cc:1918] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:46:01.233346 2694 log.cc:1196] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/wal/wals/5be4c626da9642d5b092eaa6944fa892
I20260620 14:46:01.233556 2694 ts_tablet_manager.cc:1939] T 5be4c626da9642d5b092eaa6944fa892 P 83c0797ece354bcda365f0fc36cb476e: Deleting consensus metadata
I20260620 14:46:01.233718 2696 ts_tablet_manager.cc:1918] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:46:01.233884 2696 log.cc:1196] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/5be4c626da9642d5b092eaa6944fa892
I20260620 14:46:01.233879 2098 catalog_manager.cc:4841] TS 83c0797ece354bcda365f0fc36cb476e (127.0.98.1:34179): tablet 5be4c626da9642d5b092eaa6944fa892 (table test-table [id=2f0792f2c2d745ca8cd808154493fd92]) successfully deleted
I20260620 14:46:01.234151 2696 ts_tablet_manager.cc:1939] T 5be4c626da9642d5b092eaa6944fa892 P 6a0180a125c241169a838b3aa130326e: Deleting consensus metadata
I20260620 14:46:01.234387 2695 raft_consensus.cc:2229] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:46:01.234447 2098 catalog_manager.cc:4841] TS 6a0180a125c241169a838b3aa130326e (127.0.98.2:32781): tablet 5be4c626da9642d5b092eaa6944fa892 (table test-table [id=2f0792f2c2d745ca8cd808154493fd92]) successfully deleted
I20260620 14:46:01.234484 2695 raft_consensus.cc:2258] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:01.234674 2695 ts_tablet_manager.cc:1905] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:01.236018 2695 ts_tablet_manager.cc:1918] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:46:01.236070 2695 log.cc:1196] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/5be4c626da9642d5b092eaa6944fa892
I20260620 14:46:01.236244 2695 ts_tablet_manager.cc:1939] T 5be4c626da9642d5b092eaa6944fa892 P 19b7d50b7b25407392d1009864c83f23: Deleting consensus metadata
I20260620 14:46:01.236718 2099 catalog_manager.cc:4841] TS 19b7d50b7b25407392d1009864c83f23 (127.0.98.3:36833): tablet 5be4c626da9642d5b092eaa6944fa892 (table test-table [id=2f0792f2c2d745ca8cd808154493fd92]) successfully deleted
Jun 20 14:46:01 dist-test-slave-cx89 krb5kdc[2067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966761, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:46:01 dist-test-slave-cx89 krb5kdc[2067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966761, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:01.251484 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:01.245036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.0.98.62:33065)
0620 14:46:01.245165 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:01.245167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:01.245176 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:46:01.245244 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:01.245245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:01.245256 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:01.245312 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:01.245805 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:01.246304 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:01.246997 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:01.247195 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:01.247248 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:01.249307 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:01.249308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:01.249308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:01.249316 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:01.250477 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:01.250773 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:01.250775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:01.250775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:01.250826 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:01.250978 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:01.250978 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:01.250979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:01.251084 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:01.251168 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:01.251265 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 14:46:01.251344 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260620 14:46:01.252684 2111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:46:01.252830 2111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:46:01.254096 2249 tablet_service.cc:1467] Processing CreateTablet for tablet b37202144f7b451cb742c529d9790ba3 (DEFAULT_TABLE table=test-table [id=9b2f654f08d7470b90b61cb1d5e7b70a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:01.254112 2529 tablet_service.cc:1467] Processing CreateTablet for tablet b37202144f7b451cb742c529d9790ba3 (DEFAULT_TABLE table=test-table [id=9b2f654f08d7470b90b61cb1d5e7b70a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:01.254106 2389 tablet_service.cc:1467] Processing CreateTablet for tablet b37202144f7b451cb742c529d9790ba3 (DEFAULT_TABLE table=test-table [id=9b2f654f08d7470b90b61cb1d5e7b70a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:01.254242 2529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b37202144f7b451cb742c529d9790ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:01.254242 2389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b37202144f7b451cb742c529d9790ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:01.254245 2249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b37202144f7b451cb742c529d9790ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:01.255565 2706 tablet_bootstrap.cc:492] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e: Bootstrap starting.
I20260620 14:46:01.255652 2707 tablet_bootstrap.cc:492] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23: Bootstrap starting.
I20260620 14:46:01.255844 2705 tablet_bootstrap.cc:492] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e: Bootstrap starting.
I20260620 14:46:01.255936 2706 tablet_bootstrap.cc:654] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:01.256114 2707 tablet_bootstrap.cc:654] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:01.256189 2705 tablet_bootstrap.cc:654] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:01.256677 2707 tablet_bootstrap.cc:492] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23: No bootstrap required, opened a new log
I20260620 14:46:01.256690 2706 tablet_bootstrap.cc:492] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e: No bootstrap required, opened a new log
I20260620 14:46:01.256722 2705 tablet_bootstrap.cc:492] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e: No bootstrap required, opened a new log
I20260620 14:46:01.256729 2707 ts_tablet_manager.cc:1397] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:01.256728 2706 ts_tablet_manager.cc:1397] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:01.256762 2705 ts_tablet_manager.cc:1397] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:01.256853 2706 raft_consensus.cc:348] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:46:01.256906 2706 raft_consensus.cc:374] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:01.256860 2705 raft_consensus.cc:348] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:46:01.256860 2707 raft_consensus.cc:348] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:46:01.256918 2706 raft_consensus.cc:729] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c0797ece354bcda365f0fc36cb476e, State: Initialized, Role: FOLLOWER
I20260620 14:46:01.256920 2705 raft_consensus.cc:374] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:01.256920 2707 raft_consensus.cc:374] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:01.256940 2707 raft_consensus.cc:729] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b7d50b7b25407392d1009864c83f23, State: Initialized, Role: FOLLOWER
I20260620 14:46:01.256940 2705 raft_consensus.cc:729] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0180a125c241169a838b3aa130326e, State: Initialized, Role: FOLLOWER
I20260620 14:46:01.256953 2706 consensus_queue.cc:260] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:46:01.256989 2707 consensus_queue.cc:260] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:46:01.257004 2705 consensus_queue.cc:260] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:46:01.257050 2706 ts_tablet_manager.cc:1428] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:01.257090 2705 ts_tablet_manager.cc:1428] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:01.257149 2707 ts_tablet_manager.cc:1428] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:01.461910 2675 raft_consensus.cc:482] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:46:01.462018 2675 raft_consensus.cc:504] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:46:01.462173 2675 leader_election.cc:288] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0180a125c241169a838b3aa130326e (127.0.98.2:32781), 19b7d50b7b25407392d1009864c83f23 (127.0.98.3:36833)
I20260620 14:46:01.462381 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37202144f7b451cb742c529d9790ba3" candidate_uuid: "83c0797ece354bcda365f0fc36cb476e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0180a125c241169a838b3aa130326e" is_pre_election: true
I20260620 14:46:01.462380 2549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37202144f7b451cb742c529d9790ba3" candidate_uuid: "83c0797ece354bcda365f0fc36cb476e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b7d50b7b25407392d1009864c83f23" is_pre_election: true
I20260620 14:46:01.462473 2549 raft_consensus.cc:2454] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c0797ece354bcda365f0fc36cb476e in term 0.
I20260620 14:46:01.462468 2409 raft_consensus.cc:2454] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c0797ece354bcda365f0fc36cb476e in term 0.
I20260620 14:46:01.462654 2202 leader_election.cc:302] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0180a125c241169a838b3aa130326e, 83c0797ece354bcda365f0fc36cb476e; no voters:
I20260620 14:46:01.462745 2675 raft_consensus.cc:2781] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:46:01.462785 2675 raft_consensus.cc:482] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:46:01.462798 2675 raft_consensus.cc:3037] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:01.463477 2675 raft_consensus.cc:504] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:46:01.463605 2675 leader_election.cc:288] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [CANDIDATE]: Term 1 election: Requested vote from peers 6a0180a125c241169a838b3aa130326e (127.0.98.2:32781), 19b7d50b7b25407392d1009864c83f23 (127.0.98.3:36833)
I20260620 14:46:01.463750 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37202144f7b451cb742c529d9790ba3" candidate_uuid: "83c0797ece354bcda365f0fc36cb476e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0180a125c241169a838b3aa130326e"
I20260620 14:46:01.463829 2409 raft_consensus.cc:3037] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:01.463802 2549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37202144f7b451cb742c529d9790ba3" candidate_uuid: "83c0797ece354bcda365f0fc36cb476e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b7d50b7b25407392d1009864c83f23"
I20260620 14:46:01.463881 2549 raft_consensus.cc:3037] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:01.464454 2409 raft_consensus.cc:2454] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c0797ece354bcda365f0fc36cb476e in term 1.
I20260620 14:46:01.464478 2549 raft_consensus.cc:2454] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c0797ece354bcda365f0fc36cb476e in term 1.
I20260620 14:46:01.464622 2203 leader_election.cc:302] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b7d50b7b25407392d1009864c83f23, 83c0797ece354bcda365f0fc36cb476e; no voters:
I20260620 14:46:01.464728 2675 raft_consensus.cc:2781] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:01.464778 2675 raft_consensus.cc:686] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [term 1 LEADER]: Becoming Leader. State: Replica: 83c0797ece354bcda365f0fc36cb476e, State: Running, Role: LEADER
I20260620 14:46:01.464825 2675 consensus_queue.cc:237] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:46:01.465394 2110 catalog_manager.cc:5495] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c0797ece354bcda365f0fc36cb476e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "83c0797ece354bcda365f0fc36cb476e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } health_report { overall_health: UNKNOWN } } }
I20260620 14:46:01.485628 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:01.483436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51450 (local address 127.0.98.1:34179)
0620 14:46:01.483561 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:01.483562 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:01.483569 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:46:01.483588 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:01.483588 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:01.483597 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:01.483649 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:46:01.483782 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:01.484291 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:01.485116 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:01.485303 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:01.485350 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:01.485446 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:46:01.485472 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:46:01.485500 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:01.485542 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 14:46:01.485557 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260620 14:46:01.489738 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:01.487721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41099 (local address 127.0.98.1:34179)
0620 14:46:01.487769 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:01.487771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:01.487859 (+ 88us) server_negotiation.cc:407] Connection header received
0620 14:46:01.487916 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:01.487916 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:01.487925 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:01.487973 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:46:01.488073 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:01.488503 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:01.489164 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:01.489347 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:01.489396 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:01.489499 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:46:01.489527 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:46:01.489566 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:01.489622 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 14:46:01.489654 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":30}
I20260620 14:46:01.489740 2717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:01.487674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34179 (local address 127.0.98.1:41099)
0620 14:46:01.487804 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:01.487813 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:01.487866 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:01.487976 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:01.487977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:01.487978 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:46:01.488023 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:01.488024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:01.488569 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:01.488569 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:01.489101 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:01.489103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:01.489399 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:01.489399 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:01.489447 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:01.489450 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:46:01.489572 (+ 122us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:46:01.489573 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:46:01.489619 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 14:46:01.489672 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260620 14:46:01.491605 2549 raft_consensus.cc:1261] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23 [term 1 FOLLOWER]: Refusing update from remote peer 83c0797ece354bcda365f0fc36cb476e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:01.491605 2409 raft_consensus.cc:1261] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e [term 1 FOLLOWER]: Refusing update from remote peer 83c0797ece354bcda365f0fc36cb476e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:01.491802 2675 consensus_queue.cc:1035] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:46:01.491904 2672 consensus_queue.cc:1035] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [LEADER]: Connected to new peer: Peer: permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:46:01.501645 2111 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:01.501756 2111 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:01.502311 2111 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277: Sending DeleteTablet for 3 replicas of tablet b37202144f7b451cb742c529d9790ba3
I20260620 14:46:01.502535 2529 tablet_service.cc:1514] Processing DeleteTablet for tablet b37202144f7b451cb742c529d9790ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42250
I20260620 14:46:01.502642 2389 tablet_service.cc:1514] Processing DeleteTablet for tablet b37202144f7b451cb742c529d9790ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35908
I20260620 14:46:01.502645 2249 tablet_service.cc:1514] Processing DeleteTablet for tablet b37202144f7b451cb742c529d9790ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51434
I20260620 14:46:01.502725 2696 tablet_replica.cc:330] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e: stopping tablet replica
I20260620 14:46:01.502729 2694 tablet_replica.cc:330] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e: stopping tablet replica
I20260620 14:46:01.502727 2695 tablet_replica.cc:330] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23: stopping tablet replica
I20260620 14:46:01.502772 2696 raft_consensus.cc:2229] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:46:01.502772 2695 raft_consensus.cc:2229] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:46:01.502777 2694 raft_consensus.cc:2229] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:46:01.502805 2696 raft_consensus.cc:2258] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:01.502806 2695 raft_consensus.cc:2258] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:01.502856 2694 raft_consensus.cc:2258] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:01.503158 2694 ts_tablet_manager.cc:1905] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:01.503273 2695 ts_tablet_manager.cc:1905] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:01.503506 2111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:46:01.503638 2111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:46:01.504151 2696 ts_tablet_manager.cc:1905] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:01.504922 2695 ts_tablet_manager.cc:1918] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:46:01.504969 2695 log.cc:1196] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/b37202144f7b451cb742c529d9790ba3
I20260620 14:46:01.505146 2695 ts_tablet_manager.cc:1939] T b37202144f7b451cb742c529d9790ba3 P 19b7d50b7b25407392d1009864c83f23: Deleting consensus metadata
I20260620 14:46:01.505313 2694 ts_tablet_manager.cc:1918] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:46:01.505368 2694 log.cc:1196] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-0/wal/wals/b37202144f7b451cb742c529d9790ba3
I20260620 14:46:01.505519 2696 ts_tablet_manager.cc:1918] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:46:01.505541 2694 ts_tablet_manager.cc:1939] T b37202144f7b451cb742c529d9790ba3 P 83c0797ece354bcda365f0fc36cb476e: Deleting consensus metadata
I20260620 14:46:01.505561 2696 log.cc:1196] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/b37202144f7b451cb742c529d9790ba3
I20260620 14:46:01.505721 2696 ts_tablet_manager.cc:1939] T b37202144f7b451cb742c529d9790ba3 P 6a0180a125c241169a838b3aa130326e: Deleting consensus metadata
I20260620 14:46:01.505767 2389 tablet_service.cc:1467] Processing CreateTablet for tablet 10fd05b701334988b6c2c7f65b01efdf (DEFAULT_TABLE table=test-table [id=e36a06371f474379bdcefbe7983748d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:01.505872 2389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10fd05b701334988b6c2c7f65b01efdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:01.506943 2705 tablet_bootstrap.cc:492] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e: Bootstrap starting.
I20260620 14:46:01.507179 2705 tablet_bootstrap.cc:654] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:01.507251 2098 catalog_manager.cc:4841] TS 6a0180a125c241169a838b3aa130326e (127.0.98.2:32781): tablet b37202144f7b451cb742c529d9790ba3 (table test-table [id=9b2f654f08d7470b90b61cb1d5e7b70a]) successfully deleted
I20260620 14:46:01.507543 2098 catalog_manager.cc:4841] TS 83c0797ece354bcda365f0fc36cb476e (127.0.98.1:34179): tablet b37202144f7b451cb742c529d9790ba3 (table test-table [id=9b2f654f08d7470b90b61cb1d5e7b70a]) successfully deleted
I20260620 14:46:01.507568 2529 tablet_service.cc:1467] Processing CreateTablet for tablet 10fd05b701334988b6c2c7f65b01efdf (DEFAULT_TABLE table=test-table [id=e36a06371f474379bdcefbe7983748d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:01.507663 2529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10fd05b701334988b6c2c7f65b01efdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:01.507846 2705 tablet_bootstrap.cc:492] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e: No bootstrap required, opened a new log
I20260620 14:46:01.507889 2705 ts_tablet_manager.cc:1397] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:46:01.507993 2705 raft_consensus.cc:348] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:46:01.508040 2705 raft_consensus.cc:374] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:01.508052 2705 raft_consensus.cc:729] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0180a125c241169a838b3aa130326e, State: Initialized, Role: FOLLOWER
I20260620 14:46:01.508086 2705 consensus_queue.cc:260] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:46:01.508153 2705 ts_tablet_manager.cc:1428] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:01.508757 2707 tablet_bootstrap.cc:492] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23: Bootstrap starting.
I20260620 14:46:01.508793 2249 tablet_service.cc:1467] Processing CreateTablet for tablet 10fd05b701334988b6c2c7f65b01efdf (DEFAULT_TABLE table=test-table [id=e36a06371f474379bdcefbe7983748d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:01.508891 2249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10fd05b701334988b6c2c7f65b01efdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:01.509045 2707 tablet_bootstrap.cc:654] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:01.509542 2707 tablet_bootstrap.cc:492] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23: No bootstrap required, opened a new log
I20260620 14:46:01.509582 2707 ts_tablet_manager.cc:1397] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:01.509708 2707 raft_consensus.cc:348] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:46:01.509761 2707 raft_consensus.cc:374] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:01.509778 2707 raft_consensus.cc:729] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b7d50b7b25407392d1009864c83f23, State: Initialized, Role: FOLLOWER
I20260620 14:46:01.509804 2707 consensus_queue.cc:260] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:46:01.509884 2707 ts_tablet_manager.cc:1428] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:01.510020 2706 tablet_bootstrap.cc:492] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e: Bootstrap starting.
I20260620 14:46:01.510337 2706 tablet_bootstrap.cc:654] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:01.510579 2099 catalog_manager.cc:4841] TS 19b7d50b7b25407392d1009864c83f23 (127.0.98.3:36833): tablet b37202144f7b451cb742c529d9790ba3 (table test-table [id=9b2f654f08d7470b90b61cb1d5e7b70a]) successfully deleted
I20260620 14:46:01.510885 2706 tablet_bootstrap.cc:492] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e: No bootstrap required, opened a new log
I20260620 14:46:01.510931 2706 ts_tablet_manager.cc:1397] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:01.511018 2706 raft_consensus.cc:348] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:46:01.511082 2706 raft_consensus.cc:374] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:01.511101 2706 raft_consensus.cc:729] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c0797ece354bcda365f0fc36cb476e, State: Initialized, Role: FOLLOWER
I20260620 14:46:01.511138 2706 consensus_queue.cc:260] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:46:01.511215 2706 ts_tablet_manager.cc:1428] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:01.831728 2720 raft_consensus.cc:482] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:46:01.831815 2720 raft_consensus.cc:504] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:46:01.831983 2720 leader_election.cc:288] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0180a125c241169a838b3aa130326e (127.0.98.2:32781), 19b7d50b7b25407392d1009864c83f23 (127.0.98.3:36833)
I20260620 14:46:01.832197 2549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10fd05b701334988b6c2c7f65b01efdf" candidate_uuid: "83c0797ece354bcda365f0fc36cb476e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b7d50b7b25407392d1009864c83f23" is_pre_election: true
I20260620 14:46:01.832216 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10fd05b701334988b6c2c7f65b01efdf" candidate_uuid: "83c0797ece354bcda365f0fc36cb476e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0180a125c241169a838b3aa130326e" is_pre_election: true
I20260620 14:46:01.832288 2549 raft_consensus.cc:2454] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c0797ece354bcda365f0fc36cb476e in term 0.
I20260620 14:46:01.832293 2409 raft_consensus.cc:2454] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c0797ece354bcda365f0fc36cb476e in term 0.
I20260620 14:46:01.832427 2202 leader_election.cc:302] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0180a125c241169a838b3aa130326e, 83c0797ece354bcda365f0fc36cb476e; no voters:
I20260620 14:46:01.832542 2720 raft_consensus.cc:2781] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:46:01.832592 2720 raft_consensus.cc:482] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:46:01.832607 2720 raft_consensus.cc:3037] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:01.833245 2720 raft_consensus.cc:504] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:46:01.833345 2720 leader_election.cc:288] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e [CANDIDATE]: Term 1 election: Requested vote from peers 6a0180a125c241169a838b3aa130326e (127.0.98.2:32781), 19b7d50b7b25407392d1009864c83f23 (127.0.98.3:36833)
I20260620 14:46:01.833477 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10fd05b701334988b6c2c7f65b01efdf" candidate_uuid: "83c0797ece354bcda365f0fc36cb476e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0180a125c241169a838b3aa130326e"
I20260620 14:46:01.833492 2549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10fd05b701334988b6c2c7f65b01efdf" candidate_uuid: "83c0797ece354bcda365f0fc36cb476e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b7d50b7b25407392d1009864c83f23"
I20260620 14:46:01.833547 2409 raft_consensus.cc:3037] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:01.833549 2549 raft_consensus.cc:3037] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:01.834084 2409 raft_consensus.cc:2454] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c0797ece354bcda365f0fc36cb476e in term 1.
I20260620 14:46:01.834107 2549 raft_consensus.cc:2454] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c0797ece354bcda365f0fc36cb476e in term 1.
I20260620 14:46:01.834259 2203 leader_election.cc:302] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b7d50b7b25407392d1009864c83f23, 83c0797ece354bcda365f0fc36cb476e; no voters:
I20260620 14:46:01.834367 2720 raft_consensus.cc:2781] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:01.834419 2720 raft_consensus.cc:686] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e [term 1 LEADER]: Becoming Leader. State: Replica: 83c0797ece354bcda365f0fc36cb476e, State: Running, Role: LEADER
I20260620 14:46:01.834456 2720 consensus_queue.cc:237] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260620 14:46:01.834973 2111 catalog_manager.cc:5495] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c0797ece354bcda365f0fc36cb476e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "83c0797ece354bcda365f0fc36cb476e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c0797ece354bcda365f0fc36cb476e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } health_report { overall_health: UNKNOWN } } }
I20260620 14:46:01.867165 2409 raft_consensus.cc:1261] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e [term 1 FOLLOWER]: Refusing update from remote peer 83c0797ece354bcda365f0fc36cb476e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:01.867165 2549 raft_consensus.cc:1261] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23 [term 1 FOLLOWER]: Refusing update from remote peer 83c0797ece354bcda365f0fc36cb476e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:01.867376 2680 consensus_queue.cc:1035] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0180a125c241169a838b3aa130326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:46:01.867515 2672 consensus_queue.cc:1035] T 10fd05b701334988b6c2c7f65b01efdf P 83c0797ece354bcda365f0fc36cb476e [LEADER]: Connected to new peer: Peer: permanent_uuid: "19b7d50b7b25407392d1009864c83f23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:46:01.871177 2673 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:46:03.387356 2110 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:03.387485 2110 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:03.388263 2110 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3bab7abae4d7447097cc15d754bb9277: Sending DeleteTablet for 3 replicas of tablet 10fd05b701334988b6c2c7f65b01efdf
I20260620 14:46:03.388485 2529 tablet_service.cc:1514] Processing DeleteTablet for tablet 10fd05b701334988b6c2c7f65b01efdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42250
I20260620 14:46:03.388571 2389 tablet_service.cc:1514] Processing DeleteTablet for tablet 10fd05b701334988b6c2c7f65b01efdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35908
I20260620 14:46:03.388659 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 2184
I20260620 14:46:03.388669 2249 tablet_service.cc:1514] Processing DeleteTablet for tablet 10fd05b701334988b6c2c7f65b01efdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51434
I20260620 14:46:03.388724 2748 tablet_replica.cc:330] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e: stopping tablet replica
I20260620 14:46:03.389201 2748 raft_consensus.cc:2229] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:46:03.389251 2748 raft_consensus.cc:2258] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:03.389463 2747 tablet_replica.cc:330] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23: stopping tablet replica
I20260620 14:46:03.389492 2748 ts_tablet_manager.cc:1905] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:03.389525 2747 raft_consensus.cc:2229] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:46:03.389592 2747 raft_consensus.cc:2258] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:03.389806 2747 ts_tablet_manager.cc:1905] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:03.391234 2748 ts_tablet_manager.cc:1918] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:46:03.391234 2747 ts_tablet_manager.cc:1918] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:46:03.391294 2748 log.cc:1196] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/10fd05b701334988b6c2c7f65b01efdf
I20260620 14:46:03.391294 2747 log.cc:1196] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/10fd05b701334988b6c2c7f65b01efdf
I20260620 14:46:03.391516 2748 ts_tablet_manager.cc:1939] T 10fd05b701334988b6c2c7f65b01efdf P 6a0180a125c241169a838b3aa130326e: Deleting consensus metadata
I20260620 14:46:03.391516 2747 ts_tablet_manager.cc:1939] T 10fd05b701334988b6c2c7f65b01efdf P 19b7d50b7b25407392d1009864c83f23: Deleting consensus metadata
I20260620 14:46:03.391789 2099 catalog_manager.cc:4841] TS 19b7d50b7b25407392d1009864c83f23 (127.0.98.3:36833): tablet 10fd05b701334988b6c2c7f65b01efdf (table test-table [id=e36a06371f474379bdcefbe7983748d1]) successfully deleted
I20260620 14:46:03.391803 2098 catalog_manager.cc:4841] TS 6a0180a125c241169a838b3aa130326e (127.0.98.2:32781): tablet 10fd05b701334988b6c2c7f65b01efdf (table test-table [id=e36a06371f474379bdcefbe7983748d1]) successfully deleted
W20260620 14:46:03.394934 2098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34179 (error 108)
I20260620 14:46:03.395175 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 2324
I20260620 14:46:03.395855 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:03.395258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34179 (local address 127.0.0.1:51452)
0620 14:46:03.395772 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:03.395817 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":50,"threads_started":1}
W20260620 14:46:03.395963 2098 catalog_manager.cc:4584] TS 83c0797ece354bcda365f0fc36cb476e (127.0.98.1:34179): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 10fd05b701334988b6c2c7f65b01efdf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34179: connect: Connection refused (error 111)
I20260620 14:46:03.400300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 2464
I20260620 14:46:03.404613 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 2080
2026-06-20T14:46:03Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5999 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:46:03 dist-test-slave-cx89 krb5kdc[2753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:46:03 dist-test-slave-cx89 krb5kdc[2753](info): set up 2 sockets
Jun 20 14:46:03 dist-test-slave-cx89 krb5kdc[2753](info): commencing operation
krb5kdc: starting...
W20260620 14:46:04.358866 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:46:04 dist-test-slave-cx89 krb5kdc[2753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:46:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:46:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:46:04.467989 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35391
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:04.546427 2766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:04.546635 2766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:04.546655 2766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:04.547798 2766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:46:04.547830 2766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:04.547843 2766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:04.547854 2766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:46:04.547865 2766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:46:04.549310 2766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:04.549501 2766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:04.549678 2766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:04.554340 2771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:04.554414 2766 server_base.cc:1034] running on GCE node
W20260620 14:46:04.554378 2772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:04.554378 2774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:04.554809 2766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:04.555065 2766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:04.556195 2766 hybrid_clock.cc:648] HybridClock initialized: now 1781966764556182 us; error 30 us; skew 500 ppm
Jun 20 14:46:04 dist-test-slave-cx89 krb5kdc[2753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:04.558523 2766 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:46:04.558924 2766 webserver.cc:460] Webserver started at http://127.0.98.62:46249/ using document root <none> and password file <none>
I20260620 14:46:04.559134 2766 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:04.559175 2766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:04.559298 2766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:04.560354 2766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "dc49905690474d60af1dcd0c92425ab7"
format_stamp: "Formatted at 2026-06-20 14:46:04 on dist-test-slave-cx89"
server_key: "32c2264e691a6b6ea3728056ae6c49a3"
server_key_iv: "e0059732657b54f01e509bb0ed8f1f8f"
server_key_version: "encryptionkey@0"
I20260620 14:46:04.560667 2766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "dc49905690474d60af1dcd0c92425ab7"
format_stamp: "Formatted at 2026-06-20 14:46:04 on dist-test-slave-cx89"
server_key: "32c2264e691a6b6ea3728056ae6c49a3"
server_key_iv: "e0059732657b54f01e509bb0ed8f1f8f"
server_key_version: "encryptionkey@0"
I20260620 14:46:04.561880 2766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:46:04.562549 2781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:04.562712 2766 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:04.562799 2766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "dc49905690474d60af1dcd0c92425ab7"
format_stamp: "Formatted at 2026-06-20 14:46:04 on dist-test-slave-cx89"
server_key: "32c2264e691a6b6ea3728056ae6c49a3"
server_key_iv: "e0059732657b54f01e509bb0ed8f1f8f"
server_key_version: "encryptionkey@0"
I20260620 14:46:04.562850 2766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:04.580899 2766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:04.583042 2766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:04.583155 2766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:04.586701 2766 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35391
I20260620 14:46:04.586752 2833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35391 every 8 connection(s)
I20260620 14:46:04.587184 2766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:46:04.587809 2834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:04.589982 2834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7: Bootstrap starting.
I20260620 14:46:04.590531 2834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:04.590799 2834 log.cc:824] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:04.591429 2834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7: No bootstrap required, opened a new log
I20260620 14:46:04.592303 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 2766
I20260620 14:46:04.592386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:46:04.592548 392 external_mini_cluster.cc:1428] Setting key 18e80c64433041448958aa7c84466389
I20260620 14:46:04.593750 2834 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc49905690474d60af1dcd0c92425ab7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35391 } }
I20260620 14:46:04.593895 2834 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:04.593928 2834 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc49905690474d60af1dcd0c92425ab7, State: Initialized, Role: FOLLOWER
I20260620 14:46:04.594084 2834 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc49905690474d60af1dcd0c92425ab7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35391 } }
I20260620 14:46:04.594163 2834 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:46:04.594200 2834 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:46:04.594246 2834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:04.594851 2834 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc49905690474d60af1dcd0c92425ab7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35391 } }
I20260620 14:46:04.594938 2834 leader_election.cc:302] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc49905690474d60af1dcd0c92425ab7; no voters:
I20260620 14:46:04.595077 2834 leader_election.cc:288] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:46:04.595114 2839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:04.595259 2839 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7 [term 1 LEADER]: Becoming Leader. State: Replica: dc49905690474d60af1dcd0c92425ab7, State: Running, Role: LEADER
I20260620 14:46:04.595328 2834 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:46:04.595359 2839 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc49905690474d60af1dcd0c92425ab7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35391 } }
I20260620 14:46:04.595830 2840 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc49905690474d60af1dcd0c92425ab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc49905690474d60af1dcd0c92425ab7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35391 } } }
I20260620 14:46:04.595909 2840 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:04.596036 2841 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc49905690474d60af1dcd0c92425ab7. Latest consensus state: current_term: 1 leader_uuid: "dc49905690474d60af1dcd0c92425ab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc49905690474d60af1dcd0c92425ab7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35391 } } }
I20260620 14:46:04.596091 2841 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:04.596540 2848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:46:04.597057 2848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:46:04.598821 2848 catalog_manager.cc:1347] Generated new cluster ID: 0709e0a7bd944c13885adf372801fcfb
I20260620 14:46:04.598891 2848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:46:04 dist-test-slave-cx89 krb5kdc[2753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:04.603235 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:04.593030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58830 (local address 127.0.98.62:35391)
0620 14:46:04.593210 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:04.593219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:04.593243 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:46:04.593719 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:04.593725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:04.593768 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:04.593873 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:04.594615 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:04.595773 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:04.596846 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:04.597356 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:04.597454 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:04.599792 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:04.599796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:04.599797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:04.599808 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:04.601339 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:04.601641 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:04.601644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:04.601645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:04.601745 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:04.601909 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:04.601910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:04.601911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:04.602142 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:04.602891 (+ 749us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:04.602986 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:46:04.603093 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260620 14:46:04.623759 2848 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:46:04.624329 2848 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:46:04.628172 2848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc49905690474d60af1dcd0c92425ab7: Generated new TSK 0
I20260620 14:46:04.628332 2848 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:46:04.657647 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35391
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:04.761651 2860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:04.761922 2860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:04.761958 2860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:04.763698 2860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:04.763754 2860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:04.763792 2860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:46:04.765897 2860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:04.766142 2860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:04.766390 2860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:04.771263 2865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:04.771402 2860 server_base.cc:1034] running on GCE node
W20260620 14:46:04.771358 2866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:04.771333 2868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:04.771706 2860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:04.771944 2860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:04.773079 2860 hybrid_clock.cc:648] HybridClock initialized: now 1781966764773064 us; error 31 us; skew 500 ppm
Jun 20 14:46:04 dist-test-slave-cx89 krb5kdc[2753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:04.775324 2860 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:46:04.775753 2860 webserver.cc:460] Webserver started at http://127.0.98.1:34307/ using document root <none> and password file <none>
I20260620 14:46:04.775942 2860 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:04.775987 2860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:04.776103 2860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:04.777092 2860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "94fb5a801bb149adb065926e9f13f88f"
format_stamp: "Formatted at 2026-06-20 14:46:04 on dist-test-slave-cx89"
server_key: "1d921e4d7dd90cb7d547931c6aca95bd"
server_key_iv: "5e3dac844c4c438f392e0dbc0d667e73"
server_key_version: "encryptionkey@0"
I20260620 14:46:04.777413 2860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "94fb5a801bb149adb065926e9f13f88f"
format_stamp: "Formatted at 2026-06-20 14:46:04 on dist-test-slave-cx89"
server_key: "1d921e4d7dd90cb7d547931c6aca95bd"
server_key_iv: "5e3dac844c4c438f392e0dbc0d667e73"
server_key_version: "encryptionkey@0"
I20260620 14:46:04.778571 2860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:46:04.779199 2875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:04.779350 2860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:04.779446 2860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "94fb5a801bb149adb065926e9f13f88f"
format_stamp: "Formatted at 2026-06-20 14:46:04 on dist-test-slave-cx89"
server_key: "1d921e4d7dd90cb7d547931c6aca95bd"
server_key_iv: "5e3dac844c4c438f392e0dbc0d667e73"
server_key_version: "encryptionkey@0"
I20260620 14:46:04.779507 2860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:04.800913 2860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:04.802681 2860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:04.802829 2860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:04.803049 2860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:04.803376 2860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:04.803408 2860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:04.803452 2860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:04.803475 2860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:04.808769 2860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37573
I20260620 14:46:04.808812 2988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37573 every 8 connection(s)
I20260620 14:46:04.809105 2860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:46:04.812906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 2860
I20260620 14:46:04.812978 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:46:04.813122 392 external_mini_cluster.cc:1428] Setting key 37b8346757f3269dff6db93640e0bf97
Jun 20 14:46:04 dist-test-slave-cx89 krb5kdc[2753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:04.818329 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:04.809799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49261 (local address 127.0.98.62:35391)
0620 14:46:04.809854 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:04.809856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:04.810055 (+ 199us) server_negotiation.cc:407] Connection header received
0620 14:46:04.810951 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:04.810952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:04.810961 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:04.811015 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:04.811631 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:04.812125 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:04.813102 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:04.813328 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:04.813395 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:04.815695 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:04.815697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:04.815697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:04.815703 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:04.816937 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:04.817408 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:04.817409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:04.817409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:04.817465 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:04.817725 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:04.817726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:04.817726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:04.817822 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:04.817867 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:04.818185 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 14:46:04.818266 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:46:04.818554 2991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:04.809764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35391 (local address 127.0.98.1:49261)
0620 14:46:04.809934 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:04.809978 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:04.810870 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:04.811073 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:04.811076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:04.811327 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:04.811564 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:04.811569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:04.812186 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:04.812186 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:04.812983 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:04.812986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:04.813400 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:04.813401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:04.813544 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:04.814044 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:04.814052 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:04.815587 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:04.817016 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:04.817022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:04.817025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:04.817322 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:04.817530 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:04.817531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:04.817531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:04.817667 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:04.817876 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:04.817881 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:04.818046 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 14:46:04.818171 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 14:46:04.818333 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260620 14:46:04.819067 2989 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35391
I20260620 14:46:04.819154 2989 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:04.819331 2989 heartbeater.cc:511] Master 127.0.98.62:35391 requested a full tablet report, sending...
I20260620 14:46:04.819832 2798 ts_manager.cc:194] Registered new tserver with Master: 94fb5a801bb149adb065926e9f13f88f (127.0.98.1:37573)
I20260620 14:46:04.820434 2798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49261
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:46:04.836107 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35391
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:04.913386 2994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:04.913590 2994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:04.913621 2994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:04.914749 2994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:04.914788 2994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:04.914822 2994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:46:04.916288 2994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:04.916522 2994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:04.916720 2994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:04.921365 2999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:04.921341 3002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:04.921343 3000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:04.921391 2994 server_base.cc:1034] running on GCE node
I20260620 14:46:04.921693 2994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:04.921914 2994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:04.923074 2994 hybrid_clock.cc:648] HybridClock initialized: now 1781966764923058 us; error 30 us; skew 500 ppm
Jun 20 14:46:04 dist-test-slave-cx89 krb5kdc[2753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:04.925273 2994 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:46:04.925654 2994 webserver.cc:460] Webserver started at http://127.0.98.2:44797/ using document root <none> and password file <none>
I20260620 14:46:04.925848 2994 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:04.925894 2994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:04.926009 2994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:04.926977 2994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "75402334101c48f6a92d6f901cd16c2f"
format_stamp: "Formatted at 2026-06-20 14:46:04 on dist-test-slave-cx89"
server_key: "9885c33a1ed7e9e98edffd91c1772070"
server_key_iv: "215a838f6dab30986b14a65355750dfe"
server_key_version: "encryptionkey@0"
I20260620 14:46:04.927285 2994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "75402334101c48f6a92d6f901cd16c2f"
format_stamp: "Formatted at 2026-06-20 14:46:04 on dist-test-slave-cx89"
server_key: "9885c33a1ed7e9e98edffd91c1772070"
server_key_iv: "215a838f6dab30986b14a65355750dfe"
server_key_version: "encryptionkey@0"
I20260620 14:46:04.928531 2994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:46:04.929203 3009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:04.929363 2994 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:46:04.929440 2994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "75402334101c48f6a92d6f901cd16c2f"
format_stamp: "Formatted at 2026-06-20 14:46:04 on dist-test-slave-cx89"
server_key: "9885c33a1ed7e9e98edffd91c1772070"
server_key_iv: "215a838f6dab30986b14a65355750dfe"
server_key_version: "encryptionkey@0"
I20260620 14:46:04.929498 2994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:04.945624 2994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:04.947593 2994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:04.947734 2994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:04.947944 2994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:04.948216 2994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:04.948248 2994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:04.948275 2994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:04.948293 2994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:04.953610 2994 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43015
I20260620 14:46:04.953658 3122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43015 every 8 connection(s)
I20260620 14:46:04.953946 2994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:46:04 dist-test-slave-cx89 krb5kdc[2753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:04.960347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 2994
I20260620 14:46:04.960438 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:46:04.960633 392 external_mini_cluster.cc:1428] Setting key b2afe91034fdc3c3a4f5d7bbeb5d0a5a
I20260620 14:46:04.963129 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:04.954614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47371 (local address 127.0.98.62:35391)
0620 14:46:04.954682 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:04.954684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:04.954867 (+ 183us) server_negotiation.cc:407] Connection header received
0620 14:46:04.955834 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:04.955836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:04.955846 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:04.955902 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:04.956570 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:04.957083 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:04.958088 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:04.958286 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:04.958342 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:04.960284 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:04.960285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:04.960286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:04.960294 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:04.961773 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:04.962121 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:04.962122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:04.962122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:04.962181 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:04.962507 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:04.962508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:04.962508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:04.962642 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:04.962693 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:04.963003 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 14:46:04.963066 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:46:04.963465 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:04.954588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35391 (local address 127.0.98.2:47371)
0620 14:46:04.954752 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:04.954793 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:04.955738 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:04.955967 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:04.955971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:04.956273 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:04.956479 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:04.956485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:04.957181 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:04.957182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:04.958001 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:04.958003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:04.958349 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:04.958349 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:04.958442 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:04.958862 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:04.958869 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:04.960190 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:04.961845 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:04.961850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:04.961852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:04.962060 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:04.962278 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:04.962279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:04.962280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:04.962428 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:04.962696 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:04.962701 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:04.962875 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 14:46:04.962984 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:46:04.963151 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260620 14:46:04.963949 3123 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35391
I20260620 14:46:04.964036 3123 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:04.964212 3123 heartbeater.cc:511] Master 127.0.98.62:35391 requested a full tablet report, sending...
I20260620 14:46:04.964550 2798 ts_manager.cc:194] Registered new tserver with Master: 75402334101c48f6a92d6f901cd16c2f (127.0.98.2:43015)
I20260620 14:46:04.964923 2798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47371
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:46:04.982191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35391
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:05.060273 3128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:05.060468 3128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:05.060499 3128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:05.061614 3128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:05.061651 3128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:05.061685 3128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:46:05.063140 3128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:05.063364 3128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:05.063582 3128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:05.068286 3133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:05.068266 3136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:05.068295 3134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:05.068313 3128 server_base.cc:1034] running on GCE node
I20260620 14:46:05.068642 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:05.068861 3128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:05.070006 3128 hybrid_clock.cc:648] HybridClock initialized: now 1781966765069993 us; error 30 us; skew 500 ppm
Jun 20 14:46:05 dist-test-slave-cx89 krb5kdc[2753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:05.072492 3128 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:46:05.072911 3128 webserver.cc:460] Webserver started at http://127.0.98.3:39487/ using document root <none> and password file <none>
I20260620 14:46:05.073123 3128 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:05.073170 3128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:05.073303 3128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:05.074316 3128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "7488ec11deee4c2b885d05d49fdd03cd"
format_stamp: "Formatted at 2026-06-20 14:46:05 on dist-test-slave-cx89"
server_key: "8a15366c8b2a1d44cb4bba9b471f6e1e"
server_key_iv: "76fd6cca90242c3c84ecc4a478a0caae"
server_key_version: "encryptionkey@0"
I20260620 14:46:05.075212 3128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7488ec11deee4c2b885d05d49fdd03cd"
format_stamp: "Formatted at 2026-06-20 14:46:05 on dist-test-slave-cx89"
server_key: "8a15366c8b2a1d44cb4bba9b471f6e1e"
server_key_iv: "76fd6cca90242c3c84ecc4a478a0caae"
server_key_version: "encryptionkey@0"
I20260620 14:46:05.076507 3128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:46:05.077170 3143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:05.077306 3128 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:05.077386 3128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "7488ec11deee4c2b885d05d49fdd03cd"
format_stamp: "Formatted at 2026-06-20 14:46:05 on dist-test-slave-cx89"
server_key: "8a15366c8b2a1d44cb4bba9b471f6e1e"
server_key_iv: "76fd6cca90242c3c84ecc4a478a0caae"
server_key_version: "encryptionkey@0"
I20260620 14:46:05.077454 3128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:05.090843 3128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:05.092717 3128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:05.092857 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:05.093071 3128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:05.093371 3128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:05.093407 3128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:05.093439 3128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:05.093463 3128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:05.099632 3128 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46867
I20260620 14:46:05.099668 3256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46867 every 8 connection(s)
I20260620 14:46:05.099978 3128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:46:05 dist-test-slave-cx89 krb5kdc[2753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:05.106472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 3128
I20260620 14:46:05.106539 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:46:05.106707 392 external_mini_cluster.cc:1428] Setting key a03f1c46a100376ee16190b16d354434
I20260620 14:46:05.109015 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:05.100645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49375 (local address 127.0.98.62:35391)
0620 14:46:05.100699 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:05.100701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:05.100890 (+ 189us) server_negotiation.cc:407] Connection header received
0620 14:46:05.101766 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:05.101767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:05.101777 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:05.101844 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:05.102429 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:05.102919 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:05.104006 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:05.104204 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:05.104265 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:05.106383 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:05.106385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:05.106385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:05.106391 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:05.107811 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:05.108163 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:05.108164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:05.108165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:05.108223 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:05.108429 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:05.108430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:05.108430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:05.108546 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:05.108601 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:05.108849 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 14:46:05.108930 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:46:05.109184 3259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:05.100624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35391 (local address 127.0.98.3:49375)
0620 14:46:05.100774 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:05.100814 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:05.101673 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:05.101901 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:05.101904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:05.102152 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:05.102336 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:05.102340 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:05.102996 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:05.102997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:05.103917 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:05.103920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:05.104299 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:05.104300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:05.104411 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:05.104949 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:05.104957 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:05.106288 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:05.107883 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:05.107889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:05.107892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:05.108097 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:05.108273 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:05.108274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:05.108274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:05.108375 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:05.108598 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:05.108604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:05.108735 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 14:46:05.108820 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:46:05.108970 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260620 14:46:05.109589 3257 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35391
I20260620 14:46:05.109674 3257 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:05.109814 3257 heartbeater.cc:511] Master 127.0.98.62:35391 requested a full tablet report, sending...
I20260620 14:46:05.110118 2798 ts_manager.cc:194] Registered new tserver with Master: 7488ec11deee4c2b885d05d49fdd03cd (127.0.98.3:46867)
I20260620 14:46:05.110486 2798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49375
I20260620 14:46:05.118369 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:46:05.128280 2838 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:05.127338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58832 (local address 127.0.98.62:35391)
0620 14:46:05.127412 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:05.127434 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:05.127497 (+ 63us) server_negotiation.cc:407] Connection header received
0620 14:46:05.127563 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:05.127564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:05.127574 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:05.127622 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:05.128187 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58832: BlockingRecv error: recv got EOF from 127.0.0.1:58832 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:46:05.129092 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 2860
I20260620 14:46:05.132925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 2994
I20260620 14:46:05.136396 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 3128
I20260620 14:46:05.139741 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 2766
2026-06-20T14:46:05Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1734 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:46:05 dist-test-slave-cx89 krb5kdc[3269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:46:05 dist-test-slave-cx89 krb5kdc[3269](info): set up 2 sockets
Jun 20 14:46:05 dist-test-slave-cx89 krb5kdc[3269](info): commencing operation
krb5kdc: starting...
W20260620 14:46:06.086915 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:46:06 dist-test-slave-cx89 krb5kdc[3269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:46:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:46:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:46:06.194263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34077
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:06.270699 3282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:06.270897 3282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:06.270925 3282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:06.272068 3282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:46:06.272109 3282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:06.272126 3282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:06.272143 3282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:46:06.272158 3282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:46:06.273618 3282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:06.273859 3282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:06.274056 3282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:06.278561 3290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:06.278555 3287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:06.278762 3282 server_base.cc:1034] running on GCE node
W20260620 14:46:06.278553 3288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:06.279069 3282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:06.279349 3282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:06.280493 3282 hybrid_clock.cc:648] HybridClock initialized: now 1781966766280486 us; error 42 us; skew 500 ppm
Jun 20 14:46:06 dist-test-slave-cx89 krb5kdc[3269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:06.286623 3282 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:46:06.287166 3282 webserver.cc:460] Webserver started at http://127.0.98.62:43689/ using document root <none> and password file <none>
I20260620 14:46:06.287397 3282 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:06.287482 3282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:06.287611 3282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:06.288666 3282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "bfbf144224964a158a95fa51466659a0"
format_stamp: "Formatted at 2026-06-20 14:46:06 on dist-test-slave-cx89"
server_key: "88a058b9912ef236a73d6bf9a85d8322"
server_key_iv: "0a5c10dce62e5f804bccc5132032df7a"
server_key_version: "encryptionkey@0"
I20260620 14:46:06.289000 3282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "bfbf144224964a158a95fa51466659a0"
format_stamp: "Formatted at 2026-06-20 14:46:06 on dist-test-slave-cx89"
server_key: "88a058b9912ef236a73d6bf9a85d8322"
server_key_iv: "0a5c10dce62e5f804bccc5132032df7a"
server_key_version: "encryptionkey@0"
I20260620 14:46:06.290222 3282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:46:06.290930 3297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:06.291100 3282 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:06.291181 3282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "bfbf144224964a158a95fa51466659a0"
format_stamp: "Formatted at 2026-06-20 14:46:06 on dist-test-slave-cx89"
server_key: "88a058b9912ef236a73d6bf9a85d8322"
server_key_iv: "0a5c10dce62e5f804bccc5132032df7a"
server_key_version: "encryptionkey@0"
I20260620 14:46:06.291247 3282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:06.307948 3282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:06.310043 3282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:06.310159 3282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:06.313699 3282 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34077
I20260620 14:46:06.313731 3349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34077 every 8 connection(s)
I20260620 14:46:06.314026 3282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:46:06.314836 3350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:06.317031 3350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0: Bootstrap starting.
I20260620 14:46:06.317534 3350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:06.317766 3350 log.cc:824] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:06.318346 3350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0: No bootstrap required, opened a new log
I20260620 14:46:06.318589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 3282
I20260620 14:46:06.318675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:46:06.318842 392 external_mini_cluster.cc:1428] Setting key a28a7293bb04d81c8d1741d38277a908
I20260620 14:46:06.320785 3350 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfbf144224964a158a95fa51466659a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34077 } }
I20260620 14:46:06.320951 3350 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:06.320986 3350 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfbf144224964a158a95fa51466659a0, State: Initialized, Role: FOLLOWER
I20260620 14:46:06.321130 3350 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfbf144224964a158a95fa51466659a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34077 } }
I20260620 14:46:06.321209 3350 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:46:06.321234 3350 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:46:06.321281 3350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:06.321852 3350 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfbf144224964a158a95fa51466659a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34077 } }
I20260620 14:46:06.321951 3350 leader_election.cc:302] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfbf144224964a158a95fa51466659a0; no voters:
I20260620 14:46:06.322091 3350 leader_election.cc:288] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:46:06.322145 3355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:06.322304 3355 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0 [term 1 LEADER]: Becoming Leader. State: Replica: bfbf144224964a158a95fa51466659a0, State: Running, Role: LEADER
I20260620 14:46:06.322324 3350 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:46:06.322402 3355 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfbf144224964a158a95fa51466659a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34077 } }
I20260620 14:46:06.322733 3355 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfbf144224964a158a95fa51466659a0. Latest consensus state: current_term: 1 leader_uuid: "bfbf144224964a158a95fa51466659a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfbf144224964a158a95fa51466659a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34077 } } }
I20260620 14:46:06.322820 3355 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:06.322978 3357 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfbf144224964a158a95fa51466659a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfbf144224964a158a95fa51466659a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34077 } } }
I20260620 14:46:06.323042 3357 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:06.323156 3364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:46:06.323900 3364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:46:06.326784 3364 catalog_manager.cc:1347] Generated new cluster ID: e98945087c1d4e6c8fb30885ed4e86b1
I20260620 14:46:06.326844 3364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:46:06 dist-test-slave-cx89 krb5kdc[3269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:06.330415 3354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:06.319295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.0.98.62:34077)
0620 14:46:06.319495 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:06.319505 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:06.319538 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:46:06.320755 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:06.320765 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:06.320830 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:06.320984 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:06.321631 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:06.322993 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:06.324074 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:06.324656 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:06.324765 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:06.327403 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:06.327406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:06.327407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:06.327441 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:06.328901 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:06.329211 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:06.329214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:06.329214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:06.329313 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:06.329491 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:06.329492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:06.329492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:06.329718 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:06.329854 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:06.329963 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 14:46:06.330092 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
I20260620 14:46:06.331215 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:46:06.345325 3364 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:46:06.345970 3364 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:46:06.347961 3354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:06.343951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40658 (local address 127.0.98.62:34077)
0620 14:46:06.343997 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:06.343999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:06.344120 (+ 121us) server_negotiation.cc:407] Connection header received
0620 14:46:06.344198 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:06.344199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:06.344209 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:06.344263 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:06.344628 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:06.345391 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:06.346363 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:06.346686 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:06.346757 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:06.347475 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:06.347477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:06.347477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:46:06.347699 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:06.347717 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:46:06.347724 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:06.347784 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:06.347833 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 14:46:06.347883 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:46:06.353782 3364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfbf144224964a158a95fa51466659a0: Generated new TSK 0
I20260620 14:46:06.353950 3364 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:46:06.358371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 3282
2026-06-20T14:46:06Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1216 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:46:06 dist-test-slave-cx89 krb5kdc[3383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:46:06 dist-test-slave-cx89 krb5kdc[3383](info): set up 2 sockets
Jun 20 14:46:06 dist-test-slave-cx89 krb5kdc[3383](info): commencing operation
krb5kdc: starting...
W20260620 14:46:07.311182 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:46:07 dist-test-slave-cx89 krb5kdc[3383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:46:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:46:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:46:07.420027 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36249
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:07.497506 3396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:07.497705 3396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:07.497726 3396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:07.498844 3396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:46:07.498895 3396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:07.498908 3396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:07.498919 3396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:46:07.498930 3396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:46:07.500391 3396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:07.500612 3396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:07.500813 3396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:46:07.505336 3396 server_base.cc:1034] running on GCE node
W20260620 14:46:07.505306 3401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:07.505292 3404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:07.505306 3402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:07.505710 3396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:07.505967 3396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:07.507099 3396 hybrid_clock.cc:648] HybridClock initialized: now 1781966767507086 us; error 29 us; skew 500 ppm
Jun 20 14:46:07 dist-test-slave-cx89 krb5kdc[3383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:07.509459 3396 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:46:07.509867 3396 webserver.cc:460] Webserver started at http://127.0.98.62:45577/ using document root <none> and password file <none>
I20260620 14:46:07.510038 3396 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:07.510074 3396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:07.510161 3396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:07.511125 3396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "c277832a6cef417bb4f26d5a68557651"
format_stamp: "Formatted at 2026-06-20 14:46:07 on dist-test-slave-cx89"
server_key: "3e6af9c761d4cb3d76e8c990c7573abd"
server_key_iv: "0a819a8386fe791e6ccb1735cdd30817"
server_key_version: "encryptionkey@0"
I20260620 14:46:07.511439 3396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "c277832a6cef417bb4f26d5a68557651"
format_stamp: "Formatted at 2026-06-20 14:46:07 on dist-test-slave-cx89"
server_key: "3e6af9c761d4cb3d76e8c990c7573abd"
server_key_iv: "0a819a8386fe791e6ccb1735cdd30817"
server_key_version: "encryptionkey@0"
I20260620 14:46:07.512610 3396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:46:07.513211 3411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:07.513373 3396 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:46:07.513459 3396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "c277832a6cef417bb4f26d5a68557651"
format_stamp: "Formatted at 2026-06-20 14:46:07 on dist-test-slave-cx89"
server_key: "3e6af9c761d4cb3d76e8c990c7573abd"
server_key_iv: "0a819a8386fe791e6ccb1735cdd30817"
server_key_version: "encryptionkey@0"
I20260620 14:46:07.513516 3396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:07.534364 3396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:07.536101 3396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:07.536229 3396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:07.539852 3396 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36249
I20260620 14:46:07.539885 3463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36249 every 8 connection(s)
I20260620 14:46:07.540189 3396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:46:07.541128 3464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:07.543239 3464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651: Bootstrap starting.
I20260620 14:46:07.543817 3464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:07.544032 3464 log.cc:824] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:07.544343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 3396
I20260620 14:46:07.544430 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:46:07.544586 392 external_mini_cluster.cc:1428] Setting key 1440d3ed4bfee1175cc2e3baed7d1097
I20260620 14:46:07.544695 3464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651: No bootstrap required, opened a new log
I20260620 14:46:07.546428 3464 raft_consensus.cc:348] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c277832a6cef417bb4f26d5a68557651" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36249 } }
I20260620 14:46:07.546558 3464 raft_consensus.cc:374] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:07.546579 3464 raft_consensus.cc:729] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c277832a6cef417bb4f26d5a68557651, State: Initialized, Role: FOLLOWER
I20260620 14:46:07.546713 3464 consensus_queue.cc:260] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c277832a6cef417bb4f26d5a68557651" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36249 } }
I20260620 14:46:07.546777 3464 raft_consensus.cc:388] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:46:07.546798 3464 raft_consensus.cc:482] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:46:07.546844 3464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:07.547355 3464 raft_consensus.cc:504] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c277832a6cef417bb4f26d5a68557651" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36249 } }
I20260620 14:46:07.547467 3464 leader_election.cc:302] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c277832a6cef417bb4f26d5a68557651; no voters:
I20260620 14:46:07.547611 3464 leader_election.cc:288] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:46:07.547631 3469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:07.547731 3469 raft_consensus.cc:686] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651 [term 1 LEADER]: Becoming Leader. State: Replica: c277832a6cef417bb4f26d5a68557651, State: Running, Role: LEADER
I20260620 14:46:07.547817 3469 consensus_queue.cc:237] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c277832a6cef417bb4f26d5a68557651" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36249 } }
I20260620 14:46:07.547860 3464 sys_catalog.cc:564] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:46:07.549288 3471 sys_catalog.cc:455] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c277832a6cef417bb4f26d5a68557651" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c277832a6cef417bb4f26d5a68557651" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36249 } } }
I20260620 14:46:07.549353 3471 sys_catalog.cc:458] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651 [sys.catalog]: This master's current role is: LEADER
W20260620 14:46:07.549486 3484 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:46:07.549566 3484 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:46:07.549692 3486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:46:07.549830 3486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:46:07.549505 3470 sys_catalog.cc:455] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c277832a6cef417bb4f26d5a68557651. Latest consensus state: current_term: 1 leader_uuid: "c277832a6cef417bb4f26d5a68557651" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c277832a6cef417bb4f26d5a68557651" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36249 } } }
I20260620 14:46:07.550112 3470 sys_catalog.cc:458] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:07.551154 3486 catalog_manager.cc:1347] Generated new cluster ID: 523614b183804b418c7aa64882ab2c8a
I20260620 14:46:07.551203 3486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:46:07 dist-test-slave-cx89 krb5kdc[3383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:07.555790 3468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:07.545080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44550 (local address 127.0.98.62:36249)
0620 14:46:07.545223 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:07.545232 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:07.545306 (+ 74us) server_negotiation.cc:407] Connection header received
0620 14:46:07.545812 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:07.545818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:07.545865 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:07.545990 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:07.546563 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:07.547955 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:07.548994 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:07.549557 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:07.549657 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:07.552388 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:07.552392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:07.552393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:07.552404 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:07.554187 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:07.554566 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:07.554568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:07.554569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:07.554659 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:07.554854 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:07.554854 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:07.554855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:07.555116 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:07.555245 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:07.555362 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 14:46:07.555601 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
I20260620 14:46:07.587545 3486 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:46:07.588078 3486 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:46:07.592626 3486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c277832a6cef417bb4f26d5a68557651: Generated new TSK 0
I20260620 14:46:07.592798 3486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:46:07.618422 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36249
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:07.696236 3490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:07.696435 3490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:07.696457 3490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:07.697595 3490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:07.697635 3490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:07.697664 3490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:46:07.699046 3490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:07.699262 3490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:07.699477 3490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:07.704257 3495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:07.704211 3498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:07.704238 3490 server_base.cc:1034] running on GCE node
W20260620 14:46:07.704211 3496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:07.704618 3490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:07.704854 3490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:07.705988 3490 hybrid_clock.cc:648] HybridClock initialized: now 1781966767705971 us; error 34 us; skew 500 ppm
Jun 20 14:46:07 dist-test-slave-cx89 krb5kdc[3383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:07.708369 3490 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:46:07.708756 3490 webserver.cc:460] Webserver started at http://127.0.98.1:34731/ using document root <none> and password file <none>
I20260620 14:46:07.709017 3490 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:07.709069 3490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:07.709192 3490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:07.710196 3490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "282d9d97d4ff410a899eac8ba6012615"
format_stamp: "Formatted at 2026-06-20 14:46:07 on dist-test-slave-cx89"
server_key: "ae6935576bb2836d33de962075f5fcc8"
server_key_iv: "da18448c7b4fa0e44167dfc5c4cee944"
server_key_version: "encryptionkey@0"
I20260620 14:46:07.710511 3490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "282d9d97d4ff410a899eac8ba6012615"
format_stamp: "Formatted at 2026-06-20 14:46:07 on dist-test-slave-cx89"
server_key: "ae6935576bb2836d33de962075f5fcc8"
server_key_iv: "da18448c7b4fa0e44167dfc5c4cee944"
server_key_version: "encryptionkey@0"
I20260620 14:46:07.711709 3490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:46:07.712390 3505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:07.712535 3490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:46:07.712613 3490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "282d9d97d4ff410a899eac8ba6012615"
format_stamp: "Formatted at 2026-06-20 14:46:07 on dist-test-slave-cx89"
server_key: "ae6935576bb2836d33de962075f5fcc8"
server_key_iv: "da18448c7b4fa0e44167dfc5c4cee944"
server_key_version: "encryptionkey@0"
I20260620 14:46:07.712677 3490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:07.725533 3490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:07.727560 3490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:07.727694 3490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:07.727896 3490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:07.728178 3490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:07.728209 3490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:07.728236 3490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:07.728257 3490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:07.733484 3490 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35423
I20260620 14:46:07.733538 3618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35423 every 8 connection(s)
I20260620 14:46:07.733810 3490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:46:07 dist-test-slave-cx89 krb5kdc[3383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:07.742154 3468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:07.734461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47145 (local address 127.0.98.62:36249)
0620 14:46:07.734506 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:07.734508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:07.734692 (+ 184us) server_negotiation.cc:407] Connection header received
0620 14:46:07.735191 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:07.735191 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:07.735206 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:07.735242 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:07.736192 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:07.736684 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:07.737563 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:07.737757 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:07.737800 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:07.739812 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:07.739813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:07.739813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:07.739821 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:07.741008 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:07.741408 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:07.741409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:07.741410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:07.741469 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:07.741661 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:07.741662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:07.741662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:07.741773 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:07.741821 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:07.742020 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 14:46:07.742087 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:46:07.742326 3621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:07.734439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36249 (local address 127.0.98.1:47145)
0620 14:46:07.734579 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:07.734620 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:07.735096 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:07.735304 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:07.735307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:07.735935 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:07.736122 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:07.736126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:07.736747 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:07.736747 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:07.737504 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:07.737506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:07.737811 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:07.737811 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:07.737908 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:07.738358 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:07.738366 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:07.739690 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:07.741090 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:07.741097 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:07.741098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:07.741310 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:07.741530 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:07.741530 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:07.741531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:07.741620 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:07.741816 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:07.741818 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:07.741929 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 14:46:07.742004 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:46:07.742117 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260620 14:46:07.742527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 3490
I20260620 14:46:07.742609 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:46:07.742806 392 external_mini_cluster.cc:1428] Setting key 84431f7d4198a94719f4bc0a5fdfd6e2
I20260620 14:46:07.742843 3619 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36249
I20260620 14:46:07.742936 3619 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:07.743114 3619 heartbeater.cc:511] Master 127.0.98.62:36249 requested a full tablet report, sending...
I20260620 14:46:07.743642 3428 ts_manager.cc:194] Registered new tserver with Master: 282d9d97d4ff410a899eac8ba6012615 (127.0.98.1:35423)
I20260620 14:46:07.744234 3428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47145
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:46:07.764330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36249
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:07.842213 3624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:07.842432 3624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:07.842454 3624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:07.843611 3624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:07.843653 3624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:07.843678 3624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:46:07.845074 3624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:07.845263 3624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:07.845443 3624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:07.850139 3629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:07.850188 3624 server_base.cc:1034] running on GCE node
W20260620 14:46:07.850139 3630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:07.850137 3632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:07.850471 3624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:07.850692 3624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:07.851846 3624 hybrid_clock.cc:648] HybridClock initialized: now 1781966767851831 us; error 28 us; skew 500 ppm
Jun 20 14:46:07 dist-test-slave-cx89 krb5kdc[3383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:07.854209 3624 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:46:07.854637 3624 webserver.cc:460] Webserver started at http://127.0.98.2:37795/ using document root <none> and password file <none>
I20260620 14:46:07.854852 3624 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:07.854899 3624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:07.855017 3624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:07.856072 3624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "bb340fb3dead4a04a78f0811c7cecbee"
format_stamp: "Formatted at 2026-06-20 14:46:07 on dist-test-slave-cx89"
server_key: "de7ca3c6aeef4248d1ad5f7a33239e5a"
server_key_iv: "fb760ca8c37b3855dbff89ddcbf21f7b"
server_key_version: "encryptionkey@0"
I20260620 14:46:07.856393 3624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bb340fb3dead4a04a78f0811c7cecbee"
format_stamp: "Formatted at 2026-06-20 14:46:07 on dist-test-slave-cx89"
server_key: "de7ca3c6aeef4248d1ad5f7a33239e5a"
server_key_iv: "fb760ca8c37b3855dbff89ddcbf21f7b"
server_key_version: "encryptionkey@0"
I20260620 14:46:07.857565 3624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:46:07.858212 3639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:07.858362 3624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:07.858430 3624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "bb340fb3dead4a04a78f0811c7cecbee"
format_stamp: "Formatted at 2026-06-20 14:46:07 on dist-test-slave-cx89"
server_key: "de7ca3c6aeef4248d1ad5f7a33239e5a"
server_key_iv: "fb760ca8c37b3855dbff89ddcbf21f7b"
server_key_version: "encryptionkey@0"
I20260620 14:46:07.858482 3624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:07.867268 3624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:07.869328 3624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:07.869436 3624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:07.869611 3624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:07.869859 3624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:07.869887 3624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:07.869908 3624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:07.869923 3624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:07.875234 3624 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37531
I20260620 14:46:07.875293 3752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37531 every 8 connection(s)
I20260620 14:46:07.875617 3624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:46:07.878492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 3624
I20260620 14:46:07.878582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:46:07.878784 392 external_mini_cluster.cc:1428] Setting key f45689ec84c56862fb8775501909b470
Jun 20 14:46:07 dist-test-slave-cx89 krb5kdc[3383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:07.884531 3468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:07.876256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60451 (local address 127.0.98.62:36249)
0620 14:46:07.876309 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:07.876310 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:07.876479 (+ 169us) server_negotiation.cc:407] Connection header received
0620 14:46:07.876958 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:07.876959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:07.876967 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:07.877026 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:07.877899 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:07.878531 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:07.879837 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:07.880035 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:07.880093 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:07.882055 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:07.882056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:07.882056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:07.882064 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:07.883298 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:07.883680 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:07.883682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:07.883682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:07.883734 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:07.883943 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:07.883944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:07.883944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:07.884054 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:07.884108 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:07.884360 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 14:46:07.884440 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:46:07.884711 3755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:07.876221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36249 (local address 127.0.98.2:60451)
0620 14:46:07.876371 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:07.876411 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:07.876881 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:07.877096 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:07.877099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:07.877634 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:07.877825 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:07.877830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:07.878640 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:07.878642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:07.879763 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:07.879766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:07.880104 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:07.880105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:07.880231 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:07.880706 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:07.880714 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:07.881973 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:07.883365 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:07.883371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:07.883374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:07.883618 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:07.883787 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:07.883787 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:07.883788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:07.883892 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:07.884105 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:07.884111 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:07.884246 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 14:46:07.884335 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:46:07.884485 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260620 14:46:07.885124 3753 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36249
I20260620 14:46:07.885210 3753 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:07.885378 3753 heartbeater.cc:511] Master 127.0.98.62:36249 requested a full tablet report, sending...
I20260620 14:46:07.885741 3428 ts_manager.cc:194] Registered new tserver with Master: bb340fb3dead4a04a78f0811c7cecbee (127.0.98.2:37531)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:46:07.886231 3428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:46:07.902590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36249
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:07.980669 3758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:07.980871 3758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:07.980893 3758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:07.982000 3758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:07.982034 3758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:07.982059 3758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:46:07.983455 3758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:07.983630 3758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:07.983805 3758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:07.988442 3763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:07.988428 3764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:07.988459 3758 server_base.cc:1034] running on GCE node
W20260620 14:46:07.988415 3766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:07.988798 3758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:07.989037 3758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:07.990180 3758 hybrid_clock.cc:648] HybridClock initialized: now 1781966767990163 us; error 37 us; skew 500 ppm
Jun 20 14:46:07 dist-test-slave-cx89 krb5kdc[3383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:07.992708 3758 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:46:07.993148 3758 webserver.cc:460] Webserver started at http://127.0.98.3:44423/ using document root <none> and password file <none>
I20260620 14:46:07.993350 3758 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:07.993396 3758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:07.993516 3758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:07.994509 3758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "b094f721fb8043c48f04dc516900aa6c"
format_stamp: "Formatted at 2026-06-20 14:46:07 on dist-test-slave-cx89"
server_key: "4b9ad856a445faf2989e5a909d6c7ddd"
server_key_iv: "8ff39cdbe00cfd1c3e2a683237c9396e"
server_key_version: "encryptionkey@0"
I20260620 14:46:07.994823 3758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b094f721fb8043c48f04dc516900aa6c"
format_stamp: "Formatted at 2026-06-20 14:46:07 on dist-test-slave-cx89"
server_key: "4b9ad856a445faf2989e5a909d6c7ddd"
server_key_iv: "8ff39cdbe00cfd1c3e2a683237c9396e"
server_key_version: "encryptionkey@0"
I20260620 14:46:07.996061 3758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:46:07.996747 3773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:07.996909 3758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:46:07.996987 3758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "b094f721fb8043c48f04dc516900aa6c"
format_stamp: "Formatted at 2026-06-20 14:46:07 on dist-test-slave-cx89"
server_key: "4b9ad856a445faf2989e5a909d6c7ddd"
server_key_iv: "8ff39cdbe00cfd1c3e2a683237c9396e"
server_key_version: "encryptionkey@0"
I20260620 14:46:07.997045 3758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:08.015748 3758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:08.017765 3758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:08.017904 3758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:08.018108 3758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:08.018374 3758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:08.018407 3758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:08.018437 3758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:08.018461 3758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:08.023792 3758 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36523
I20260620 14:46:08.023865 3886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36523 every 8 connection(s)
I20260620 14:46:08.024194 3758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:46:08.026983 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 3758
I20260620 14:46:08.027081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:46:08.027277 392 external_mini_cluster.cc:1428] Setting key 61b0f27c8e6fd0d8b2b470bab74657f7
Jun 20 14:46:08 dist-test-slave-cx89 krb5kdc[3383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:08.033700 3468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:08.025221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37081 (local address 127.0.98.62:36249)
0620 14:46:08.025279 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:08.025281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:08.025518 (+ 237us) server_negotiation.cc:407] Connection header received
0620 14:46:08.026202 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:08.026203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:08.026216 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:08.026266 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:08.027443 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:08.028097 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:08.029048 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:08.029247 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:08.029301 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:08.031238 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:08.031239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:08.031239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:08.031247 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:08.032526 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:08.032864 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:08.032865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:08.032865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:08.032920 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:08.033122 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:08.033123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:08.033123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:08.033234 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:08.033287 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:08.033537 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 14:46:08.033611 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:46:08.033874 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:08.025205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36249 (local address 127.0.98.3:37081)
0620 14:46:08.025379 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:08.025427 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:08.026060 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:08.026346 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:08.026350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:08.027070 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:08.027340 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:08.027347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:08.028177 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:08.028178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:08.028975 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:08.028977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:08.029310 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:08.029311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:08.029437 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:08.029882 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:08.029897 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:08.031159 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:08.032595 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:08.032602 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:08.032604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:08.032804 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:08.032971 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:08.032972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:08.032972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:08.033070 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:08.033286 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:08.033292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:08.033426 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 14:46:08.033512 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:46:08.033663 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260620 14:46:08.034286 3887 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36249
I20260620 14:46:08.034438 3887 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:08.034627 3887 heartbeater.cc:511] Master 127.0.98.62:36249 requested a full tablet report, sending...
I20260620 14:46:08.035025 3428 ts_manager.cc:194] Registered new tserver with Master: b094f721fb8043c48f04dc516900aa6c (127.0.98.3:36523)
I20260620 14:46:08.035382 3428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37081
I20260620 14:46:08.038816 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:46:08 dist-test-slave-cx89 krb5kdc[3383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966768, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 14:46:08 dist-test-slave-cx89 krb5kdc[3383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966768, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:08.060709 3468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:08.053728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44560 (local address 127.0.98.62:36249)
0620 14:46:08.053778 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:08.053780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:08.053875 (+ 95us) server_negotiation.cc:407] Connection header received
0620 14:46:08.053924 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:08.053925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:08.053933 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:08.053999 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:08.054511 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:08.055004 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:08.055773 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:08.055961 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:08.056024 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:08.058310 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:08.058312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:08.058312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:08.058319 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:08.059628 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:08.059945 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:08.059945 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:08.059946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:08.059991 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:08.060116 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:08.060117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:08.060117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:08.060256 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:08.060331 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:08.060570 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 14:46:08.060640 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 14:46:08.060948 3428 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44560
I20260620 14:46:08.061338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 3490
I20260620 14:46:08.065135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 3624
I20260620 14:46:08.068629 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 3758
I20260620 14:46:08.072116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 3396
2026-06-20T14:46:08Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1715 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:46:08 dist-test-slave-cx89 krb5kdc[3899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:46:08 dist-test-slave-cx89 krb5kdc[3899](info): set up 2 sockets
Jun 20 14:46:08 dist-test-slave-cx89 krb5kdc[3899](info): commencing operation
krb5kdc: starting...
W20260620 14:46:09.019001 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:46:09 dist-test-slave-cx89 krb5kdc[3899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:46:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:46:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:46:09.126633 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43803
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:09.203680 3912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:09.203900 3912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:09.203922 3912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:09.205039 3912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:46:09.205070 3912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:09.205083 3912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:09.205094 3912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:46:09.205106 3912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:46:09.206476 3912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:09.206667 3912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:09.206830 3912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:09.211251 3920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:09.211282 3918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:09.211266 3917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:09.211370 3912 server_base.cc:1034] running on GCE node
I20260620 14:46:09.211753 3912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:09.212035 3912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:09.213202 3912 hybrid_clock.cc:648] HybridClock initialized: now 1781966769213189 us; error 28 us; skew 500 ppm
Jun 20 14:46:09 dist-test-slave-cx89 krb5kdc[3899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:09.215571 3912 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:46:09.215970 3912 webserver.cc:460] Webserver started at http://127.0.98.62:34551/ using document root <none> and password file <none>
I20260620 14:46:09.216166 3912 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:09.216212 3912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:09.216349 3912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:09.217344 3912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "aad8c8c78b46492389d6468af9c8a45d"
format_stamp: "Formatted at 2026-06-20 14:46:09 on dist-test-slave-cx89"
server_key: "e0821620d3e4e6bfa0089a795b89c0e0"
server_key_iv: "c4db13609f87098b1093915a4fcc27c9"
server_key_version: "encryptionkey@0"
I20260620 14:46:09.217686 3912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "aad8c8c78b46492389d6468af9c8a45d"
format_stamp: "Formatted at 2026-06-20 14:46:09 on dist-test-slave-cx89"
server_key: "e0821620d3e4e6bfa0089a795b89c0e0"
server_key_iv: "c4db13609f87098b1093915a4fcc27c9"
server_key_version: "encryptionkey@0"
I20260620 14:46:09.218828 3912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:46:09.219496 3927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:09.219641 3912 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:09.219719 3912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "aad8c8c78b46492389d6468af9c8a45d"
format_stamp: "Formatted at 2026-06-20 14:46:09 on dist-test-slave-cx89"
server_key: "e0821620d3e4e6bfa0089a795b89c0e0"
server_key_iv: "c4db13609f87098b1093915a4fcc27c9"
server_key_version: "encryptionkey@0"
I20260620 14:46:09.219786 3912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:09.229635 3912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:09.231699 3912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:09.231829 3912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:09.235316 3912 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43803
I20260620 14:46:09.235347 3979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43803 every 8 connection(s)
I20260620 14:46:09.235671 3912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:46:09.236423 3980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:09.238528 3980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d: Bootstrap starting.
I20260620 14:46:09.239074 3980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:09.239333 3980 log.cc:824] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:09.239949 3980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d: No bootstrap required, opened a new log
I20260620 14:46:09.240850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 3912
I20260620 14:46:09.240938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:46:09.241070 392 external_mini_cluster.cc:1428] Setting key caa83c0af9cecc958a22b05371a3eaca
I20260620 14:46:09.241664 3980 raft_consensus.cc:348] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad8c8c78b46492389d6468af9c8a45d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43803 } }
I20260620 14:46:09.241835 3980 raft_consensus.cc:374] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:09.241879 3980 raft_consensus.cc:729] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad8c8c78b46492389d6468af9c8a45d, State: Initialized, Role: FOLLOWER
I20260620 14:46:09.241997 3980 consensus_queue.cc:260] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad8c8c78b46492389d6468af9c8a45d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43803 } }
I20260620 14:46:09.242070 3980 raft_consensus.cc:388] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:46:09.242131 3980 raft_consensus.cc:482] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:46:09.242180 3980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:09.243094 3980 raft_consensus.cc:504] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad8c8c78b46492389d6468af9c8a45d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43803 } }
I20260620 14:46:09.243201 3980 leader_election.cc:302] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aad8c8c78b46492389d6468af9c8a45d; no voters:
I20260620 14:46:09.243350 3980 leader_election.cc:288] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:46:09.243394 3985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:09.243561 3985 raft_consensus.cc:686] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d [term 1 LEADER]: Becoming Leader. State: Replica: aad8c8c78b46492389d6468af9c8a45d, State: Running, Role: LEADER
I20260620 14:46:09.243592 3980 sys_catalog.cc:564] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:46:09.243697 3985 consensus_queue.cc:237] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad8c8c78b46492389d6468af9c8a45d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43803 } }
I20260620 14:46:09.244234 3987 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aad8c8c78b46492389d6468af9c8a45d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad8c8c78b46492389d6468af9c8a45d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43803 } } }
I20260620 14:46:09.244489 3987 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:09.244426 3986 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d [sys.catalog]: SysCatalogTable state changed. Reason: New leader aad8c8c78b46492389d6468af9c8a45d. Latest consensus state: current_term: 1 leader_uuid: "aad8c8c78b46492389d6468af9c8a45d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad8c8c78b46492389d6468af9c8a45d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43803 } } }
I20260620 14:46:09.245028 4000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:46:09.245023 3986 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:09.245548 4000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:46:09.247033 4000 catalog_manager.cc:1347] Generated new cluster ID: 35dbde2470654a71aee556413f72f26e
I20260620 14:46:09.247083 4000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:46:09 dist-test-slave-cx89 krb5kdc[3899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:09.250824 3984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:09.241561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59888 (local address 127.0.98.62:43803)
0620 14:46:09.241771 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:09.241781 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:09.241813 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:46:09.241920 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:09.241927 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:09.241979 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:09.242116 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:09.242703 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:09.244046 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:09.245075 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:09.245625 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:09.245724 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:09.248066 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:09.248070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:09.248071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:09.248081 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:09.249525 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:09.249831 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:09.249834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:09.249834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:09.249924 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:09.250088 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:09.250088 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:09.250089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:09.250317 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:09.250450 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:09.250548 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 14:46:09.250672 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":108,"threads_started":1}
I20260620 14:46:09.255585 4000 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:46:09.256081 4000 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:46:09.273005 4000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aad8c8c78b46492389d6468af9c8a45d: Generated new TSK 0
I20260620 14:46:09.273219 4000 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:46:09.304381 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43803
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:09.406457 4006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:09.406658 4006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:09.406682 4006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:09.407814 4006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:09.407853 4006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:09.407884 4006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:46:09.409415 4006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:09.409597 4006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:09.409765 4006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:09.414425 4011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:09.414572 4006 server_base.cc:1034] running on GCE node
W20260620 14:46:09.414535 4014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:09.414439 4012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:09.414887 4006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:09.415089 4006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:09.416234 4006 hybrid_clock.cc:648] HybridClock initialized: now 1781966769416218 us; error 26 us; skew 500 ppm
Jun 20 14:46:09 dist-test-slave-cx89 krb5kdc[3899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:09.418684 4006 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:46:09.419081 4006 webserver.cc:460] Webserver started at http://127.0.98.1:43355/ using document root <none> and password file <none>
I20260620 14:46:09.419266 4006 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:09.419312 4006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:09.419440 4006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:09.420456 4006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "f8661747893348ffad48fad4155b73b0"
format_stamp: "Formatted at 2026-06-20 14:46:09 on dist-test-slave-cx89"
server_key: "0cf0b0453431a4923f731d5edf2ee3c9"
server_key_iv: "9011b515438186137d8214392adcf3a3"
server_key_version: "encryptionkey@0"
I20260620 14:46:09.420773 4006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f8661747893348ffad48fad4155b73b0"
format_stamp: "Formatted at 2026-06-20 14:46:09 on dist-test-slave-cx89"
server_key: "0cf0b0453431a4923f731d5edf2ee3c9"
server_key_iv: "9011b515438186137d8214392adcf3a3"
server_key_version: "encryptionkey@0"
I20260620 14:46:09.421947 4006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:46:09.422598 4021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:09.422757 4006 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:09.422842 4006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "f8661747893348ffad48fad4155b73b0"
format_stamp: "Formatted at 2026-06-20 14:46:09 on dist-test-slave-cx89"
server_key: "0cf0b0453431a4923f731d5edf2ee3c9"
server_key_iv: "9011b515438186137d8214392adcf3a3"
server_key_version: "encryptionkey@0"
I20260620 14:46:09.422909 4006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:09.432677 4006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:09.434487 4006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:09.434598 4006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:09.434844 4006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:09.435132 4006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:09.435164 4006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:09.435192 4006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:09.435209 4006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:09.440454 4006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37625
I20260620 14:46:09.440508 4134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37625 every 8 connection(s)
I20260620 14:46:09.440804 4006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:46:09 dist-test-slave-cx89 krb5kdc[3899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:09.449407 3984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:09.441475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41617 (local address 127.0.98.62:43803)
0620 14:46:09.441529 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:09.441531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:09.441707 (+ 176us) server_negotiation.cc:407] Connection header received
0620 14:46:09.442603 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:09.442604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:09.442618 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:09.442677 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:09.443361 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:09.443923 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:09.444963 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:09.445190 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:09.445253 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:09.447227 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:09.447228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:09.447228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:09.447236 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:09.448319 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:09.448690 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:09.448690 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:09.448691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:09.448733 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:09.448916 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:09.448916 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:09.448917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:09.449006 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:09.449048 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:09.449278 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 14:46:09.449331 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:46:09.449455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 4006
I20260620 14:46:09.449548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:46:09.449730 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:09.441425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43803 (local address 127.0.98.1:41617)
0620 14:46:09.441579 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:09.441621 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:09.442496 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:09.442741 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:09.442744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:09.443053 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:09.443254 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:09.443259 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:09.444026 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:09.444027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:09.444881 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:09.444883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:09.445266 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:09.445268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:09.445370 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:09.445826 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:09.445834 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:09.447142 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:09.448405 (+ 1263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:09.448411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:09.448413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:09.448627 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:09.448775 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:09.448775 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:09.448775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:09.448875 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:09.449046 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:09.449049 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:09.449158 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 14:46:09.449269 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:46:09.449508 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260620 14:46:09.449744 392 external_mini_cluster.cc:1428] Setting key 26da9a6f1e1b8eb815593774f504c9e3
I20260620 14:46:09.450276 4135 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43803
I20260620 14:46:09.450384 4135 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:09.450551 4135 heartbeater.cc:511] Master 127.0.98.62:43803 requested a full tablet report, sending...
I20260620 14:46:09.451079 3944 ts_manager.cc:194] Registered new tserver with Master: f8661747893348ffad48fad4155b73b0 (127.0.98.1:37625)
I20260620 14:46:09.451820 3944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41617
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:46:09.471623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43803
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:09.548388 4140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:09.548578 4140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:09.548609 4140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:09.549707 4140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:09.549744 4140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:09.549778 4140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:46:09.551700 4140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:09.551946 4140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:09.552155 4140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:09.556746 4145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:09.556847 4148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:09.556933 4146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:09.556945 4140 server_base.cc:1034] running on GCE node
I20260620 14:46:09.557164 4140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:09.557387 4140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:09.558521 4140 hybrid_clock.cc:648] HybridClock initialized: now 1781966769558509 us; error 31 us; skew 500 ppm
Jun 20 14:46:09 dist-test-slave-cx89 krb5kdc[3899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:09.560820 4140 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:46:09.561237 4140 webserver.cc:460] Webserver started at http://127.0.98.2:39327/ using document root <none> and password file <none>
I20260620 14:46:09.561429 4140 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:09.561466 4140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:09.561554 4140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:09.562506 4140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "8c080a1af05947e59722a25840cce026"
format_stamp: "Formatted at 2026-06-20 14:46:09 on dist-test-slave-cx89"
server_key: "a93514b53078d0b0ddfe52c411d20645"
server_key_iv: "3bfce8a1b3a80a2ee1aa9d5553b224b7"
server_key_version: "encryptionkey@0"
I20260620 14:46:09.562814 4140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8c080a1af05947e59722a25840cce026"
format_stamp: "Formatted at 2026-06-20 14:46:09 on dist-test-slave-cx89"
server_key: "a93514b53078d0b0ddfe52c411d20645"
server_key_iv: "3bfce8a1b3a80a2ee1aa9d5553b224b7"
server_key_version: "encryptionkey@0"
I20260620 14:46:09.564025 4140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:46:09.564692 4155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:09.564879 4140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:09.564956 4140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "8c080a1af05947e59722a25840cce026"
format_stamp: "Formatted at 2026-06-20 14:46:09 on dist-test-slave-cx89"
server_key: "a93514b53078d0b0ddfe52c411d20645"
server_key_iv: "3bfce8a1b3a80a2ee1aa9d5553b224b7"
server_key_version: "encryptionkey@0"
I20260620 14:46:09.565007 4140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:09.576349 4140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:09.578086 4140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:09.578189 4140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:09.578359 4140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:09.578589 4140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:09.578616 4140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:09.578636 4140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:09.578652 4140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:09.583897 4140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41863
I20260620 14:46:09.583940 4268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41863 every 8 connection(s)
I20260620 14:46:09.584213 4140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:46:09.585446 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 4140
I20260620 14:46:09.585525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:46:09.585690 392 external_mini_cluster.cc:1428] Setting key 831f3e9f1a52fa9af7d478ee3bf82c6f
Jun 20 14:46:09 dist-test-slave-cx89 krb5kdc[3899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:46:09.593140 3984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:09.584856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46257 (local address 127.0.98.62:43803)
0620 14:46:09.584907 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:09.584909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:09.585066 (+ 157us) server_negotiation.cc:407] Connection header received
0620 14:46:09.586038 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:09.586039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:09.586050 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:09.586090 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:09.586696 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:09.587184 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:09.588219 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:09.588501 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:09.588568 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:09.590572 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:09.590574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:09.590574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:09.590583 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:09.591955 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:09.592311 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:09.592312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:09.592313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:09.592365 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:09.592558 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:09.592559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:09.592560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:09.592686 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:09.592749 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:09.592977 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 14:46:09.593054 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:46:09.593281 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:09.584822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43803 (local address 127.0.98.2:46257)
0620 14:46:09.584954 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:09.584992 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:09.585947 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:09.586168 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:09.586172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:09.586426 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:09.586606 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:09.586611 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:09.587295 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:09.587297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:09.588140 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:09.588142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:09.588581 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:09.588582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:09.588688 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:09.589160 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:09.589168 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:09.590487 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:09.592024 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:09.592029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:09.592031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:09.592235 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:09.592423 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:09.592423 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:09.592424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:09.592506 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:09.592737 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:09.592740 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:09.592857 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 14:46:09.592940 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:46:09.593082 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
I20260620 14:46:09.593684 4269 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43803
I20260620 14:46:09.593775 4269 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:09.593956 4269 heartbeater.cc:511] Master 127.0.98.62:43803 requested a full tablet report, sending...
I20260620 14:46:09.594283 3944 ts_manager.cc:194] Registered new tserver with Master: 8c080a1af05947e59722a25840cce026 (127.0.98.2:41863)
I20260620 14:46:09.594615 3944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:46:09.608521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43803
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:09.685333 4274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:09.685528 4274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:09.685549 4274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:09.686647 4274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:09.686686 4274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:09.686712 4274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:46:09.688163 4274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:09.688380 4274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:09.688581 4274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:09.693078 4282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:09.693140 4279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:09.693181 4274 server_base.cc:1034] running on GCE node
W20260620 14:46:09.693295 4280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:09.693491 4274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:09.693719 4274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:09.694854 4274 hybrid_clock.cc:648] HybridClock initialized: now 1781966769694838 us; error 27 us; skew 500 ppm
Jun 20 14:46:09 dist-test-slave-cx89 krb5kdc[3899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:09.697129 4274 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:46:09.697530 4274 webserver.cc:460] Webserver started at http://127.0.98.3:41055/ using document root <none> and password file <none>
I20260620 14:46:09.697721 4274 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:09.697765 4274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:09.697891 4274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:09.698828 4274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "203a97b7fce540f6a536dc619ed8f827"
format_stamp: "Formatted at 2026-06-20 14:46:09 on dist-test-slave-cx89"
server_key: "73e3e3613df9d96113e99d941d7a93fa"
server_key_iv: "fcacbf2e163880c2be7652e56bd8f5cf"
server_key_version: "encryptionkey@0"
I20260620 14:46:09.699151 4274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "203a97b7fce540f6a536dc619ed8f827"
format_stamp: "Formatted at 2026-06-20 14:46:09 on dist-test-slave-cx89"
server_key: "73e3e3613df9d96113e99d941d7a93fa"
server_key_iv: "fcacbf2e163880c2be7652e56bd8f5cf"
server_key_version: "encryptionkey@0"
I20260620 14:46:09.700356 4274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:46:09.700999 4289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:09.701135 4274 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:46:09.701211 4274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "203a97b7fce540f6a536dc619ed8f827"
format_stamp: "Formatted at 2026-06-20 14:46:09 on dist-test-slave-cx89"
server_key: "73e3e3613df9d96113e99d941d7a93fa"
server_key_iv: "fcacbf2e163880c2be7652e56bd8f5cf"
server_key_version: "encryptionkey@0"
I20260620 14:46:09.701274 4274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:09.711546 4274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:09.713471 4274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:09.713594 4274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:09.713791 4274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:09.714073 4274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:09.714104 4274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:09.714133 4274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:09.714156 4274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:09.719194 4274 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42545
I20260620 14:46:09.719264 4402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42545 every 8 connection(s)
I20260620 14:46:09.719576 4274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:46:09.722741 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 4274
I20260620 14:46:09.722829 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:46:09.722992 392 external_mini_cluster.cc:1428] Setting key 59c9c94b17d3f34b39c3b7be3750b9d0
Jun 20 14:46:09 dist-test-slave-cx89 krb5kdc[3899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:09.728295 3984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:09.720212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43081 (local address 127.0.98.62:43803)
0620 14:46:09.720254 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:09.720256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:09.720423 (+ 167us) server_negotiation.cc:407] Connection header received
0620 14:46:09.721343 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:09.721345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:09.721357 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:09.721406 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:09.722041 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:09.722579 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:09.723886 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:09.724082 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:09.724133 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:09.726082 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:09.726083 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:09.726084 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:09.726091 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:09.727108 (+ 1017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:09.727461 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:09.727462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:09.727463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:09.727518 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:09.727724 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:09.727725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:09.727725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:09.727835 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:09.727890 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:09.728135 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 14:46:09.728211 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:46:09.728454 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:09.720193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43803 (local address 127.0.98.3:43081)
0620 14:46:09.720317 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:09.720351 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:09.721239 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:09.721477 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:09.721481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:09.721750 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:09.721944 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:09.721949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:09.722655 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:09.722656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:09.723819 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:09.723822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:09.724143 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:09.724144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:09.724274 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:09.724728 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:09.724736 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:09.726001 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:09.727174 (+ 1173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:09.727180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:09.727182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:09.727380 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:09.727565 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:09.727566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:09.727567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:09.727670 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:09.727887 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:09.727891 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:09.728022 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 14:46:09.728107 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:46:09.728249 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":60,"threads_started":1}
I20260620 14:46:09.728837 4403 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43803
I20260620 14:46:09.728916 4403 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:09.729048 4403 heartbeater.cc:511] Master 127.0.98.62:43803 requested a full tablet report, sending...
I20260620 14:46:09.729365 3944 ts_manager.cc:194] Registered new tserver with Master: 203a97b7fce540f6a536dc619ed8f827 (127.0.98.3:42545)
I20260620 14:46:09.729687 3944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43081
I20260620 14:46:09.734371 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:46:09 dist-test-slave-cx89 krb5kdc[3899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:46:09 dist-test-slave-cx89 krb5kdc[3899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:46:09.754626 4410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:09.747824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53536 (local address 127.0.98.1:37625)
0620 14:46:09.747972 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:09.747976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:09.747987 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:46:09.748010 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:09.748012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:09.748025 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:09.748086 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:09.748557 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:09.749247 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:09.749901 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:09.750195 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:09.750241 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:09.752400 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:09.752408 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:09.752408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:09.752415 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:09.753779 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:09.754045 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:09.754047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:09.754047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:09.754095 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:09.754209 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:09.754209 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:09.754209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:09.754322 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:09.754428 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:09.754460 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 14:46:09.754529 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
Jun 20 14:46:09 dist-test-slave-cx89 krb5kdc[3899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:09.761329 3984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:09.755753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.0.98.62:43803)
0620 14:46:09.755809 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:09.755812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:09.755819 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:46:09.755871 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:09.755872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:09.755881 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:09.755925 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:09.756329 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:09.756797 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:09.757416 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:09.757596 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:09.757645 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:09.759431 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:09.759433 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:09.759433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:09.759441 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:09.760530 (+ 1089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:09.760802 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:09.760802 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:09.760803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:09.760843 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:09.760962 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:09.760963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:09.760963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:09.761083 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:09.761150 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:09.761212 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 14:46:09.761264 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 14:46:09.761569 3944 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59894
I20260620 14:46:09.761854 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 4006
I20260620 14:46:09.765957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 4140
I20260620 14:46:09.769467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 4274
I20260620 14:46:09.773028 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 3912
2026-06-20T14:46:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1700 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:46:09 dist-test-slave-cx89 krb5kdc[4415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:46:09 dist-test-slave-cx89 krb5kdc[4415](info): set up 2 sockets
Jun 20 14:46:09 dist-test-slave-cx89 krb5kdc[4415](info): commencing operation
krb5kdc: starting...
W20260620 14:46:10.718995 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:46:10 dist-test-slave-cx89 krb5kdc[4415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:46:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:46:10.827380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38927
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:10.905937 4428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:10.906137 4428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:10.906158 4428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:10.907294 4428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:46:10.907335 4428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:10.907348 4428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:10.907358 4428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:46:10.907369 4428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:46:10.908870 4428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:10.909096 4428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:10.909291 4428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:10.914057 4436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:10.914063 4434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:10.914213 4428 server_base.cc:1034] running on GCE node
W20260620 14:46:10.914075 4433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:10.914487 4428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:10.914786 4428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:10.915927 4428 hybrid_clock.cc:648] HybridClock initialized: now 1781966770915914 us; error 29 us; skew 500 ppm
Jun 20 14:46:10 dist-test-slave-cx89 krb5kdc[4415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:10.918197 4428 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:46:10.918496 4428 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:10.918545 4428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:10.918660 4428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:10.919715 4428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "d4c1f93313d94de4b4983807ef368951"
format_stamp: "Formatted at 2026-06-20 14:46:10 on dist-test-slave-cx89"
server_key: "51de90aca163d10ded21c8f3a1031a01"
server_key_iv: "857eb3e78ce49799130613d86e09c390"
server_key_version: "encryptionkey@0"
I20260620 14:46:10.920027 4428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "d4c1f93313d94de4b4983807ef368951"
format_stamp: "Formatted at 2026-06-20 14:46:10 on dist-test-slave-cx89"
server_key: "51de90aca163d10ded21c8f3a1031a01"
server_key_iv: "857eb3e78ce49799130613d86e09c390"
server_key_version: "encryptionkey@0"
I20260620 14:46:10.921186 4428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:10.921867 4442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:10.922036 4428 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:10.922114 4428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "d4c1f93313d94de4b4983807ef368951"
format_stamp: "Formatted at 2026-06-20 14:46:10 on dist-test-slave-cx89"
server_key: "51de90aca163d10ded21c8f3a1031a01"
server_key_iv: "857eb3e78ce49799130613d86e09c390"
server_key_version: "encryptionkey@0"
I20260620 14:46:10.922170 4428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:10.933813 4428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:10.936053 4428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:10.936183 4428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:10.939656 4428 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38927
I20260620 14:46:10.939693 4494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38927 every 8 connection(s)
I20260620 14:46:10.939941 4428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:46:10.940783 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:10.941390 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 4428
I20260620 14:46:10.941486 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:46:10.941653 392 external_mini_cluster.cc:1428] Setting key 7bf4ba868b49fb27c70be2d98b29302b
I20260620 14:46:10.943217 4495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951: Bootstrap starting.
I20260620 14:46:10.943825 4495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:10.944137 4495 log.cc:824] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:10.944866 4495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951: No bootstrap required, opened a new log
I20260620 14:46:10.946949 4495 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4c1f93313d94de4b4983807ef368951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38927 } }
I20260620 14:46:10.947108 4495 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:10.947140 4495 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4c1f93313d94de4b4983807ef368951, State: Initialized, Role: FOLLOWER
I20260620 14:46:10.947254 4495 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4c1f93313d94de4b4983807ef368951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38927 } }
I20260620 14:46:10.947324 4495 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:46:10.947359 4495 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:46:10.947402 4495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:10.948005 4495 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4c1f93313d94de4b4983807ef368951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38927 } }
I20260620 14:46:10.948101 4495 leader_election.cc:302] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4c1f93313d94de4b4983807ef368951; no voters:
I20260620 14:46:10.948254 4495 leader_election.cc:288] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:46:10.948280 4500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:10.948423 4500 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951 [term 1 LEADER]: Becoming Leader. State: Replica: d4c1f93313d94de4b4983807ef368951, State: Running, Role: LEADER
I20260620 14:46:10.948501 4495 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:46:10.948544 4500 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4c1f93313d94de4b4983807ef368951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38927 } }
I20260620 14:46:10.948897 4500 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4c1f93313d94de4b4983807ef368951. Latest consensus state: current_term: 1 leader_uuid: "d4c1f93313d94de4b4983807ef368951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4c1f93313d94de4b4983807ef368951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38927 } } }
I20260620 14:46:10.949105 4500 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:10.949047 4502 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4c1f93313d94de4b4983807ef368951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4c1f93313d94de4b4983807ef368951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38927 } } }
I20260620 14:46:10.949450 4502 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951 [sys.catalog]: This master's current role is: LEADER
Jun 20 14:46:10 dist-test-slave-cx89 krb5kdc[4415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260620 14:46:10.950173 4516 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:46:10.950212 4516 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:46:10.950253 4509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:46:10.950394 4509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:46:10.951920 4509 catalog_manager.cc:1347] Generated new cluster ID: 87c88e8e54f8464cb99fbd6eb90c1a7d
I20260620 14:46:10.951968 4509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:46:10.953740 4498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:10.942138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.0.98.62:38927)
0620 14:46:10.943529 (+ 1391us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:10.943540 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:10.943574 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:46:10.944118 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:10.944127 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:10.944187 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:10.944333 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:10.945000 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:10.946325 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:10.947014 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:10.947625 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:10.947734 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:10.950341 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:10.950345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:10.950345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:10.950356 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:10.952236 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:10.952611 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:10.952614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:10.952614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:10.952715 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:10.952894 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:10.952895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:10.952895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:10.953188 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:10.953336 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:10.953405 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:46:10.953537 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1339,"thread_start_us":72,"threads_started":1}
I20260620 14:46:10.960294 4509 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:46:10.961004 4509 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:46:10.972559 4509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951: Generated new TSK 0
I20260620 14:46:10.972756 4509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:46:11.001647 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38927
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:11.080699 4521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:11.080910 4521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:11.080941 4521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:11.082064 4521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:11.082103 4521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:11.082139 4521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:46:11.083613 4521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:11.083832 4521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:11.084031 4521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:11.089085 4526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:11.089107 4529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:11.089097 4527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:11.089334 4521 server_base.cc:1034] running on GCE node
I20260620 14:46:11.089551 4521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:11.089766 4521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:11.090898 4521 hybrid_clock.cc:648] HybridClock initialized: now 1781966771090881 us; error 31 us; skew 500 ppm
Jun 20 14:46:11 dist-test-slave-cx89 krb5kdc[4415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:11.093194 4521 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:46:11.093488 4521 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:11.093535 4521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:11.093650 4521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:11.094645 4521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "0a160ca8c7914074ba5828309c9b3234"
format_stamp: "Formatted at 2026-06-20 14:46:11 on dist-test-slave-cx89"
server_key: "e2c32e0a09caa75b7d0a25c486c0eeeb"
server_key_iv: "255e273bfb0d0a884a165653e8b8855e"
server_key_version: "encryptionkey@0"
I20260620 14:46:11.094954 4521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0a160ca8c7914074ba5828309c9b3234"
format_stamp: "Formatted at 2026-06-20 14:46:11 on dist-test-slave-cx89"
server_key: "e2c32e0a09caa75b7d0a25c486c0eeeb"
server_key_iv: "255e273bfb0d0a884a165653e8b8855e"
server_key_version: "encryptionkey@0"
I20260620 14:46:11.096105 4521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:46:11.096745 4535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:11.096877 4521 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:11.096953 4521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "0a160ca8c7914074ba5828309c9b3234"
format_stamp: "Formatted at 2026-06-20 14:46:11 on dist-test-slave-cx89"
server_key: "e2c32e0a09caa75b7d0a25c486c0eeeb"
server_key_iv: "255e273bfb0d0a884a165653e8b8855e"
server_key_version: "encryptionkey@0"
I20260620 14:46:11.097009 4521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:11.115223 4521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:11.117201 4521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:11.117336 4521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:11.117558 4521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:11.117846 4521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:11.117877 4521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:11.117906 4521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:11.117930 4521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:11.123164 4521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33515
I20260620 14:46:11.123212 4648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33515 every 8 connection(s)
I20260620 14:46:11.123479 4521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:46:11.126250 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 4521
I20260620 14:46:11.126350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:46:11.126516 392 external_mini_cluster.cc:1428] Setting key c8e9042023e08d7157200feeaceac4c1
Jun 20 14:46:11 dist-test-slave-cx89 krb5kdc[4415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:11.132303 4498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.124086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48497 (local address 127.0.98.62:38927)
0620 14:46:11.124127 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:11.124128 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:11.124326 (+ 198us) server_negotiation.cc:407] Connection header received
0620 14:46:11.124840 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:11.124842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:11.124853 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:11.124900 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:11.125817 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.126313 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.127307 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.127552 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.127605 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:11.129803 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:11.129805 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:11.129805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:11.129813 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:11.131067 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:11.131441 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:11.131442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:11.131443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:11.131496 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:11.131704 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:11.131705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:11.131706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:11.131814 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:11.131871 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:11.132120 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 14:46:11.132213 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:46:11.132459 4651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.124062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38927 (local address 127.0.98.1:48497)
0620 14:46:11.124218 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:11.124252 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:11.124745 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:11.124985 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:11.124989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:11.125240 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:11.125735 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:11.125742 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.126390 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.126392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:11.127233 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:11.127235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.127628 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.127629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:11.127755 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:11.128248 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:11.128259 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:11.129717 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:11.131135 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:11.131142 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:11.131144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:11.131360 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:11.131547 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:11.131548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:11.131548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:11.131652 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:11.131866 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:11.131873 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:11.132008 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 14:46:11.132096 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:46:11.132246 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260620 14:46:11.133114 4649 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38927
I20260620 14:46:11.133200 4649 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:11.133342 4649 heartbeater.cc:511] Master 127.0.98.62:38927 requested a full tablet report, sending...
I20260620 14:46:11.133904 4459 ts_manager.cc:194] Registered new tserver with Master: 0a160ca8c7914074ba5828309c9b3234 (127.0.98.1:33515)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:46:11.134757 4459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:46:11.149541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38927
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:11.227159 4654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:11.227353 4654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:11.227376 4654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:11.228534 4654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:11.228577 4654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:11.228602 4654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:46:11.230059 4654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:11.230276 4654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:11.230473 4654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:11.235105 4660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:11.235188 4659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:11.235213 4654 server_base.cc:1034] running on GCE node
W20260620 14:46:11.235154 4662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:11.235558 4654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:11.235781 4654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:11.236913 4654 hybrid_clock.cc:648] HybridClock initialized: now 1781966771236903 us; error 30 us; skew 500 ppm
Jun 20 14:46:11 dist-test-slave-cx89 krb5kdc[4415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:11.239202 4654 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:46:11.239522 4654 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:11.239571 4654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:11.239693 4654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:11.240658 4654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "268f71ed200f4ffbb94455bdf94ee03c"
format_stamp: "Formatted at 2026-06-20 14:46:11 on dist-test-slave-cx89"
server_key: "25e57f93285c5e015e1b30e1f7288761"
server_key_iv: "99f3dbce90de755222ae9f1b8542eed9"
server_key_version: "encryptionkey@0"
I20260620 14:46:11.240976 4654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "268f71ed200f4ffbb94455bdf94ee03c"
format_stamp: "Formatted at 2026-06-20 14:46:11 on dist-test-slave-cx89"
server_key: "25e57f93285c5e015e1b30e1f7288761"
server_key_iv: "99f3dbce90de755222ae9f1b8542eed9"
server_key_version: "encryptionkey@0"
I20260620 14:46:11.242115 4654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:46:11.242736 4668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:11.242877 4654 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:11.242972 4654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "268f71ed200f4ffbb94455bdf94ee03c"
format_stamp: "Formatted at 2026-06-20 14:46:11 on dist-test-slave-cx89"
server_key: "25e57f93285c5e015e1b30e1f7288761"
server_key_iv: "99f3dbce90de755222ae9f1b8542eed9"
server_key_version: "encryptionkey@0"
I20260620 14:46:11.243027 4654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:11.258430 4654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:11.260205 4654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:11.260330 4654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:11.260532 4654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:11.260818 4654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:11.260848 4654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:11.260876 4654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:11.260898 4654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:11.266034 4654 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35195
I20260620 14:46:11.266083 4781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35195 every 8 connection(s)
I20260620 14:46:11.266314 4654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:46:11 dist-test-slave-cx89 krb5kdc[4415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:11.273862 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 4654
I20260620 14:46:11.273944 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:46:11.274125 392 external_mini_cluster.cc:1428] Setting key 0fcf55b90276742b74311acbdd02ad4b
I20260620 14:46:11.274559 4498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.266941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40775 (local address 127.0.98.62:38927)
0620 14:46:11.267012 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:11.267015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:11.267161 (+ 146us) server_negotiation.cc:407] Connection header received
0620 14:46:11.267632 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:11.267633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:11.267642 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:11.267697 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:11.268561 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.269074 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.269970 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.270143 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.270184 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:11.272119 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:11.272120 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:11.272120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:11.272126 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:11.273145 (+ 1019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:11.273481 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:11.273483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:11.273483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:11.273538 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:11.273758 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:11.273759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:11.273759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:11.273897 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:11.273988 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:11.274292 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 14:46:11.274393 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:46:11.274587 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.266908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38927 (local address 127.0.98.2:40775)
0620 14:46:11.267049 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:11.267084 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:11.267547 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:11.267760 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:11.267764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:11.268019 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:11.268495 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:11.268501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.269135 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.269136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:11.269913 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:11.269915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.270192 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.270192 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:11.270278 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:11.270726 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:11.270733 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:11.272040 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:11.273203 (+ 1163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:11.273207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:11.273209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:11.273404 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:11.273591 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:11.273592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:11.273592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:11.273675 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:11.273977 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:11.273983 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:11.274139 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 14:46:11.274252 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:46:11.274390 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260620 14:46:11.275049 4782 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38927
I20260620 14:46:11.275136 4782 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:11.275278 4782 heartbeater.cc:511] Master 127.0.98.62:38927 requested a full tablet report, sending...
I20260620 14:46:11.275637 4459 ts_manager.cc:194] Registered new tserver with Master: 268f71ed200f4ffbb94455bdf94ee03c (127.0.98.2:35195)
I20260620 14:46:11.276000 4459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40775
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:46:11.296205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38927
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:11.372997 4787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:11.373190 4787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:11.373212 4787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:11.374327 4787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:11.374367 4787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:11.374392 4787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:46:11.375847 4787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:11.376022 4787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:11.376192 4787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:46:11.380800 4787 server_base.cc:1034] running on GCE node
W20260620 14:46:11.380764 4795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:11.380759 4793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:11.380759 4792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:11.381114 4787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:11.381317 4787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:11.382464 4787 hybrid_clock.cc:648] HybridClock initialized: now 1781966771382447 us; error 27 us; skew 500 ppm
Jun 20 14:46:11 dist-test-slave-cx89 krb5kdc[4415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:11.388273 4787 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:46:11.388582 4787 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:11.388634 4787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:11.388751 4787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:11.389740 4787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "c177a40f10aa4bd58ec5c8448c9541e4"
format_stamp: "Formatted at 2026-06-20 14:46:11 on dist-test-slave-cx89"
server_key: "3fa051c87172fb3796d5bda76da66a74"
server_key_iv: "ad74633f66d3ca72beca7995817cc427"
server_key_version: "encryptionkey@0"
I20260620 14:46:11.390061 4787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c177a40f10aa4bd58ec5c8448c9541e4"
format_stamp: "Formatted at 2026-06-20 14:46:11 on dist-test-slave-cx89"
server_key: "3fa051c87172fb3796d5bda76da66a74"
server_key_iv: "ad74633f66d3ca72beca7995817cc427"
server_key_version: "encryptionkey@0"
I20260620 14:46:11.391227 4787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:46:11.391907 4801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:11.392048 4787 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:11.392136 4787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "c177a40f10aa4bd58ec5c8448c9541e4"
format_stamp: "Formatted at 2026-06-20 14:46:11 on dist-test-slave-cx89"
server_key: "3fa051c87172fb3796d5bda76da66a74"
server_key_iv: "ad74633f66d3ca72beca7995817cc427"
server_key_version: "encryptionkey@0"
I20260620 14:46:11.392201 4787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:11.400686 4787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:11.402361 4787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:11.402491 4787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:11.402671 4787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:11.402930 4787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:11.402961 4787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:11.402987 4787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:11.403012 4787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:11.408309 4787 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43977
I20260620 14:46:11.408361 4914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43977 every 8 connection(s)
I20260620 14:46:11.408596 4787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:46:11.410321 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 4787
I20260620 14:46:11.410427 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:46:11.410596 392 external_mini_cluster.cc:1428] Setting key 158a7be25b58d11dbcff978d478c405e
Jun 20 14:46:11 dist-test-slave-cx89 krb5kdc[4415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:11.416774 4498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.409219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59517 (local address 127.0.98.62:38927)
0620 14:46:11.409281 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:11.409282 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:11.409487 (+ 205us) server_negotiation.cc:407] Connection header received
0620 14:46:11.409962 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:11.409963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:11.409976 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:11.410027 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:11.410985 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.411542 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.412478 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.412679 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.412730 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:11.414584 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:11.414585 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:11.414586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:11.414593 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:11.415655 (+ 1062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:11.416046 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:11.416047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:11.416047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:11.416093 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:11.416270 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:11.416270 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:11.416270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:11.416361 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:11.416407 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:11.416643 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 14:46:11.416713 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:46:11.416970 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.409202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38927 (local address 127.0.98.3:59517)
0620 14:46:11.409374 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:11.409408 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:11.409865 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:11.410094 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:11.410097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:11.410342 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:11.410900 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:11.410906 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.411618 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.411619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:11.412403 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:11.412405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.412742 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.412743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:11.412857 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:11.413265 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:11.413273 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:11.414499 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:11.415757 (+ 1258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:11.415764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:11.415766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:11.415989 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:11.416138 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:11.416139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:11.416139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:11.416229 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:11.416403 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:11.416406 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:11.416521 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 14:46:11.416626 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 14:46:11.416762 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260620 14:46:11.417332 4915 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38927
I20260620 14:46:11.417416 4915 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:11.417526 4915 heartbeater.cc:511] Master 127.0.98.62:38927 requested a full tablet report, sending...
I20260620 14:46:11.417791 4459 ts_manager.cc:194] Registered new tserver with Master: c177a40f10aa4bd58ec5c8448c9541e4 (127.0.98.3:43977)
I20260620 14:46:11.418136 4459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59517
I20260620 14:46:11.422113 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:46:11.428411 4498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.423289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58826 (local address 127.0.98.62:38927)
0620 14:46:11.423335 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:11.423337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:11.423453 (+ 116us) server_negotiation.cc:407] Connection header received
0620 14:46:11.423529 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:11.423530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:11.423538 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:11.423580 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:11.424087 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.424590 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.425273 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.425486 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.425533 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:11.426535 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:11.426535 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:11.426536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:11.426540 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:11.427575 (+ 1035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:11.427884 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:11.427884 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:11.427885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:11.427926 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:11.428040 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:11.428041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:11.428041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:11.428154 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:11.428221 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:11.428287 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:46:11.428339 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:46:11.429848 4459 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:46:11.431305 4459 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:46:11.437033 4923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.433132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33515 (local address 127.0.0.1:58512)
0620 14:46:11.433467 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:11.433511 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:11.433582 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:11.433855 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:11.433856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:11.433858 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:46:11.433910 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:11.433912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.434827 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.434828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:11.435991 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:11.435993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.436744 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.436745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:11.436824 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:11.436836 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 14:46:11.436897 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:46:11.436947 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":64,"threads_started":1}
I20260620 14:46:11.437186 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.433334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43977 (local address 127.0.0.1:42928)
0620 14:46:11.433682 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:11.433690 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:11.433729 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:11.433952 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:11.433952 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:11.433955 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:46:11.433986 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:11.433988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.435225 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.435226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:11.436103 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:11.436105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.436882 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.436883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:11.437028 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:11.437034 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:46:11.437086 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:46:11.437120 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":29,"thread_start_us":73,"threads_started":1}
I20260620 14:46:11.437322 4926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.433283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58512 (local address 127.0.98.1:33515)
0620 14:46:11.433457 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:11.433462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:11.433566 (+ 104us) server_negotiation.cc:407] Connection header received
0620 14:46:11.433661 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:11.433663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:11.433679 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:11.433752 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:46:11.433987 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.434740 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.436082 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.436676 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.436735 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:11.436793 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:11.437205 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 14:46:11.437246 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":76,"threads_started":1}
I20260620 14:46:11.437422 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.433137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35195 (local address 127.0.0.1:56848)
0620 14:46:11.433595 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:11.433603 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:11.433651 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:11.433814 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:11.433817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:11.433819 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:46:11.433904 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:11.433906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.435208 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.435210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:11.436325 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:11.436328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.437188 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.437189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:11.437259 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:11.437262 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:46:11.437308 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 14:46:11.437347 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":29,"thread_start_us":54,"threads_started":1}
I20260620 14:46:11.437463 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.433229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56848 (local address 127.0.98.2:35195)
0620 14:46:11.433525 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:11.433529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:11.433647 (+ 118us) server_negotiation.cc:407] Connection header received
0620 14:46:11.433724 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:11.433726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:11.433741 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:11.433803 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:46:11.433983 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.435113 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.436399 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.437048 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.437103 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:11.437157 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:11.437349 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 14:46:11.437392 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":88,"threads_started":1}
I20260620 14:46:11.437911 4583 tablet_service.cc:1467] Processing CreateTablet for tablet 30fa1bf3086a4803ac9e106cf7e47f31 (DEFAULT_TABLE table=test-table [id=8a44f1b2187443a6a009c801ada4e917]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:11.438256 4583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30fa1bf3086a4803ac9e106cf7e47f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:11.439940 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.433435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42928 (local address 127.0.98.3:43977)
0620 14:46:11.433587 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:11.433591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:11.433731 (+ 140us) server_negotiation.cc:407] Connection header received
0620 14:46:11.433789 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:11.433791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:11.433806 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:11.433882 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:46:11.434053 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.435133 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.436171 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.436801 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.436881 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:11.436935 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:11.439816 (+ 2881us) server_negotiation.cc:299] Negotiation successful
0620 14:46:11.439864 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260620 14:46:11.440124 4716 tablet_service.cc:1467] Processing CreateTablet for tablet 30fa1bf3086a4803ac9e106cf7e47f31 (DEFAULT_TABLE table=test-table [id=8a44f1b2187443a6a009c801ada4e917]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:11.440423 4716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30fa1bf3086a4803ac9e106cf7e47f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:11.440577 4849 tablet_service.cc:1467] Processing CreateTablet for tablet 30fa1bf3086a4803ac9e106cf7e47f31 (DEFAULT_TABLE table=test-table [id=8a44f1b2187443a6a009c801ada4e917]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:11.440873 4849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30fa1bf3086a4803ac9e106cf7e47f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:11.441336 4929 tablet_bootstrap.cc:492] T 30fa1bf3086a4803ac9e106cf7e47f31 P 0a160ca8c7914074ba5828309c9b3234: Bootstrap starting.
I20260620 14:46:11.441861 4929 tablet_bootstrap.cc:654] T 30fa1bf3086a4803ac9e106cf7e47f31 P 0a160ca8c7914074ba5828309c9b3234: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:11.442317 4929 log.cc:824] T 30fa1bf3086a4803ac9e106cf7e47f31 P 0a160ca8c7914074ba5828309c9b3234: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:11.442966 4929 tablet_bootstrap.cc:492] T 30fa1bf3086a4803ac9e106cf7e47f31 P 0a160ca8c7914074ba5828309c9b3234: No bootstrap required, opened a new log
I20260620 14:46:11.443025 4929 ts_tablet_manager.cc:1397] T 30fa1bf3086a4803ac9e106cf7e47f31 P 0a160ca8c7914074ba5828309c9b3234: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:46:11.443607 4931 tablet_bootstrap.cc:492] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c: Bootstrap starting.
I20260620 14:46:11.443868 4932 tablet_bootstrap.cc:492] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4: Bootstrap starting.
I20260620 14:46:11.444399 4932 tablet_bootstrap.cc:654] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:11.444662 4931 tablet_bootstrap.cc:654] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:11.444913 4932 log.cc:824] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:11.445009 4929 raft_consensus.cc:348] T 30fa1bf3086a4803ac9e106cf7e47f31 P 0a160ca8c7914074ba5828309c9b3234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f71ed200f4ffbb94455bdf94ee03c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } } peers { permanent_uuid: "0a160ca8c7914074ba5828309c9b3234" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33515 } } peers { permanent_uuid: "c177a40f10aa4bd58ec5c8448c9541e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43977 } }
I20260620 14:46:11.445204 4929 raft_consensus.cc:374] T 30fa1bf3086a4803ac9e106cf7e47f31 P 0a160ca8c7914074ba5828309c9b3234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:11.445219 4931 log.cc:824] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:11.445240 4929 raft_consensus.cc:729] T 30fa1bf3086a4803ac9e106cf7e47f31 P 0a160ca8c7914074ba5828309c9b3234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a160ca8c7914074ba5828309c9b3234, State: Initialized, Role: FOLLOWER
I20260620 14:46:11.445362 4929 consensus_queue.cc:260] T 30fa1bf3086a4803ac9e106cf7e47f31 P 0a160ca8c7914074ba5828309c9b3234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f71ed200f4ffbb94455bdf94ee03c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } } peers { permanent_uuid: "0a160ca8c7914074ba5828309c9b3234" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33515 } } peers { permanent_uuid: "c177a40f10aa4bd58ec5c8448c9541e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43977 } }
I20260620 14:46:11.445658 4929 ts_tablet_manager.cc:1428] T 30fa1bf3086a4803ac9e106cf7e47f31 P 0a160ca8c7914074ba5828309c9b3234: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:46:11.445693 4932 tablet_bootstrap.cc:492] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4: No bootstrap required, opened a new log
I20260620 14:46:11.445811 4649 heartbeater.cc:503] Master 127.0.98.62:38927 was elected leader, sending a full tablet report...
I20260620 14:46:11.445894 4931 tablet_bootstrap.cc:492] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c: No bootstrap required, opened a new log
I20260620 14:46:11.445956 4931 ts_tablet_manager.cc:1397] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:46:11.446075 4932 ts_tablet_manager.cc:1397] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:46:11.447543 4932 raft_consensus.cc:348] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f71ed200f4ffbb94455bdf94ee03c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } } peers { permanent_uuid: "0a160ca8c7914074ba5828309c9b3234" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33515 } } peers { permanent_uuid: "c177a40f10aa4bd58ec5c8448c9541e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43977 } }
I20260620 14:46:11.447557 4931 raft_consensus.cc:348] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f71ed200f4ffbb94455bdf94ee03c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } } peers { permanent_uuid: "0a160ca8c7914074ba5828309c9b3234" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33515 } } peers { permanent_uuid: "c177a40f10aa4bd58ec5c8448c9541e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43977 } }
I20260620 14:46:11.447667 4931 raft_consensus.cc:374] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:11.447661 4932 raft_consensus.cc:374] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:11.447686 4931 raft_consensus.cc:729] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 268f71ed200f4ffbb94455bdf94ee03c, State: Initialized, Role: FOLLOWER
I20260620 14:46:11.447690 4932 raft_consensus.cc:729] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c177a40f10aa4bd58ec5c8448c9541e4, State: Initialized, Role: FOLLOWER
I20260620 14:46:11.447795 4932 consensus_queue.cc:260] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f71ed200f4ffbb94455bdf94ee03c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } } peers { permanent_uuid: "0a160ca8c7914074ba5828309c9b3234" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33515 } } peers { permanent_uuid: "c177a40f10aa4bd58ec5c8448c9541e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43977 } }
I20260620 14:46:11.447815 4931 consensus_queue.cc:260] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f71ed200f4ffbb94455bdf94ee03c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } } peers { permanent_uuid: "0a160ca8c7914074ba5828309c9b3234" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33515 } } peers { permanent_uuid: "c177a40f10aa4bd58ec5c8448c9541e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43977 } }
I20260620 14:46:11.448052 4932 ts_tablet_manager.cc:1428] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:46:11.448062 4915 heartbeater.cc:503] Master 127.0.98.62:38927 was elected leader, sending a full tablet report...
I20260620 14:46:11.448081 4931 ts_tablet_manager.cc:1428] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:46:11.448096 4782 heartbeater.cc:503] Master 127.0.98.62:38927 was elected leader, sending a full tablet report...
W20260620 14:46:11.516795 4783 tablet.cc:2369] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:46:11.577430 4936 raft_consensus.cc:482] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:46:11.577564 4936 raft_consensus.cc:504] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f71ed200f4ffbb94455bdf94ee03c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } } peers { permanent_uuid: "0a160ca8c7914074ba5828309c9b3234" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33515 } } peers { permanent_uuid: "c177a40f10aa4bd58ec5c8448c9541e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43977 } }
I20260620 14:46:11.577870 4936 leader_election.cc:288] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 268f71ed200f4ffbb94455bdf94ee03c (127.0.98.2:35195), 0a160ca8c7914074ba5828309c9b3234 (127.0.98.1:33515)
I20260620 14:46:11.580502 4926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.578154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45603 (local address 127.0.98.1:33515)
0620 14:46:11.578202 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:11.578204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:11.578212 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:46:11.578240 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:11.578240 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:11.578251 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:11.578310 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:46:11.578459 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.578917 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.579830 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.580198 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.580258 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:11.580268 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:11.580399 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 14:46:11.580442 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:46:11.580502 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.578034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33515 (local address 127.0.98.3:45603)
0620 14:46:11.578088 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:11.578100 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:11.578142 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:11.578324 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:11.578325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:11.578327 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:46:11.578390 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:11.578391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.578990 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.578991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:11.579745 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:11.579747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.580268 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.580269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:11.580340 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:11.580343 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:46:11.580386 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 14:46:11.580426 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:46:11.580597 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.578108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37657 (local address 127.0.98.2:35195)
0620 14:46:11.578152 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:11.578153 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:11.578307 (+ 154us) server_negotiation.cc:407] Connection header received
0620 14:46:11.578352 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:11.578353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:11.578365 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:11.578409 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:46:11.578549 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.578992 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.579940 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.580292 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.580348 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:11.580357 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:11.580497 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 14:46:11.580531 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:46:11.580623 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.578039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35195 (local address 127.0.98.3:37657)
0620 14:46:11.578253 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:11.578259 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:11.578299 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:11.578421 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:11.578422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:11.578424 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:46:11.578491 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:11.578493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.579059 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.579060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:11.579860 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:11.579862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.580361 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.580362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:11.580434 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:11.580437 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:46:11.580480 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 14:46:11.580517 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":57,"thread_start_us":99,"threads_started":1}
I20260620 14:46:11.580832 4603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fa1bf3086a4803ac9e106cf7e47f31" candidate_uuid: "c177a40f10aa4bd58ec5c8448c9541e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a160ca8c7914074ba5828309c9b3234" is_pre_election: true
I20260620 14:46:11.580930 4736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fa1bf3086a4803ac9e106cf7e47f31" candidate_uuid: "c177a40f10aa4bd58ec5c8448c9541e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "268f71ed200f4ffbb94455bdf94ee03c" is_pre_election: true
I20260620 14:46:11.580984 4603 raft_consensus.cc:2454] T 30fa1bf3086a4803ac9e106cf7e47f31 P 0a160ca8c7914074ba5828309c9b3234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c177a40f10aa4bd58ec5c8448c9541e4 in term 0.
I20260620 14:46:11.581070 4736 raft_consensus.cc:2454] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c177a40f10aa4bd58ec5c8448c9541e4 in term 0.
I20260620 14:46:11.581195 4805 leader_election.cc:302] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a160ca8c7914074ba5828309c9b3234, c177a40f10aa4bd58ec5c8448c9541e4; no voters:
I20260620 14:46:11.581331 4936 raft_consensus.cc:2781] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:46:11.581388 4936 raft_consensus.cc:482] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:46:11.581413 4936 raft_consensus.cc:3037] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:11.582072 4936 raft_consensus.cc:504] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f71ed200f4ffbb94455bdf94ee03c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } } peers { permanent_uuid: "0a160ca8c7914074ba5828309c9b3234" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33515 } } peers { permanent_uuid: "c177a40f10aa4bd58ec5c8448c9541e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43977 } }
I20260620 14:46:11.582187 4936 leader_election.cc:288] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [CANDIDATE]: Term 1 election: Requested vote from peers 268f71ed200f4ffbb94455bdf94ee03c (127.0.98.2:35195), 0a160ca8c7914074ba5828309c9b3234 (127.0.98.1:33515)
I20260620 14:46:11.582320 4736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fa1bf3086a4803ac9e106cf7e47f31" candidate_uuid: "c177a40f10aa4bd58ec5c8448c9541e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "268f71ed200f4ffbb94455bdf94ee03c"
I20260620 14:46:11.582340 4603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fa1bf3086a4803ac9e106cf7e47f31" candidate_uuid: "c177a40f10aa4bd58ec5c8448c9541e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a160ca8c7914074ba5828309c9b3234"
I20260620 14:46:11.582402 4603 raft_consensus.cc:3037] T 30fa1bf3086a4803ac9e106cf7e47f31 P 0a160ca8c7914074ba5828309c9b3234 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:11.582392 4736 raft_consensus.cc:3037] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:11.583046 4736 raft_consensus.cc:2454] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c177a40f10aa4bd58ec5c8448c9541e4 in term 1.
I20260620 14:46:11.583119 4603 raft_consensus.cc:2454] T 30fa1bf3086a4803ac9e106cf7e47f31 P 0a160ca8c7914074ba5828309c9b3234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c177a40f10aa4bd58ec5c8448c9541e4 in term 1.
I20260620 14:46:11.583212 4804 leader_election.cc:302] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 268f71ed200f4ffbb94455bdf94ee03c, c177a40f10aa4bd58ec5c8448c9541e4; no voters:
I20260620 14:46:11.583307 4936 raft_consensus.cc:2781] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:11.583467 4936 raft_consensus.cc:686] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [term 1 LEADER]: Becoming Leader. State: Replica: c177a40f10aa4bd58ec5c8448c9541e4, State: Running, Role: LEADER
I20260620 14:46:11.583549 4936 consensus_queue.cc:237] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f71ed200f4ffbb94455bdf94ee03c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } } peers { permanent_uuid: "0a160ca8c7914074ba5828309c9b3234" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33515 } } peers { permanent_uuid: "c177a40f10aa4bd58ec5c8448c9541e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43977 } }
I20260620 14:46:11.584283 4457 catalog_manager.cc:5495] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c177a40f10aa4bd58ec5c8448c9541e4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c177a40f10aa4bd58ec5c8448c9541e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "268f71ed200f4ffbb94455bdf94ee03c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a160ca8c7914074ba5828309c9b3234" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c177a40f10aa4bd58ec5c8448c9541e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43977 } health_report { overall_health: HEALTHY } } }
I20260620 14:46:11.616663 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.614451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42934 (local address 127.0.98.3:43977)
0620 14:46:11.614502 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:11.614503 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:11.614511 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:46:11.614554 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:11.614554 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:11.614574 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:11.614632 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:46:11.614748 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.615200 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.615989 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:11.616215 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:11.616264 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:11.616396 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:46:11.616501 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:46:11.616558 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:11.616572 (+ 14us) server_negotiation.cc:299] Negotiation successful
0620 14:46:11.616588 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:46:11.618376 4603 raft_consensus.cc:1261] T 30fa1bf3086a4803ac9e106cf7e47f31 P 0a160ca8c7914074ba5828309c9b3234 [term 1 FOLLOWER]: Refusing update from remote peer c177a40f10aa4bd58ec5c8448c9541e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:11.618376 4736 raft_consensus.cc:1261] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c [term 1 FOLLOWER]: Refusing update from remote peer c177a40f10aa4bd58ec5c8448c9541e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:11.618690 4939 consensus_queue.cc:1035] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "268f71ed200f4ffbb94455bdf94ee03c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:46:11.618820 4936 consensus_queue.cc:1035] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a160ca8c7914074ba5828309c9b3234" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:46:11.621433 4945 mvcc.cc:205] Tried to move back new op lower bound from 7298935896546332672 to 7298935896406810624. Current Snapshot: MvccSnapshot[applied={T|T < 7298935896546332672}]
I20260620 14:46:11.623576 4458 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:11.623663 4458 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:11.624398 4458 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4c1f93313d94de4b4983807ef368951: Sending DeleteTablet for 3 replicas of tablet 30fa1bf3086a4803ac9e106cf7e47f31
I20260620 14:46:11.624699 4583 tablet_service.cc:1514] Processing DeleteTablet for tablet 30fa1bf3086a4803ac9e106cf7e47f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58512
I20260620 14:46:11.624735 4716 tablet_service.cc:1514] Processing DeleteTablet for tablet 30fa1bf3086a4803ac9e106cf7e47f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56848
I20260620 14:46:11.624840 4849 tablet_service.cc:1514] Processing DeleteTablet for tablet 30fa1bf3086a4803ac9e106cf7e47f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42928
I20260620 14:46:11.625058 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 4521
I20260620 14:46:11.627359 4953 tablet_replica.cc:330] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4: stopping tablet replica
I20260620 14:46:11.627485 4953 raft_consensus.cc:2229] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:46:11.627606 4953 raft_consensus.cc:2258] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:11.627964 4953 ts_tablet_manager.cc:1905] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:11.629377 4953 ts_tablet_manager.cc:1918] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:46:11.629439 4953 log.cc:1196] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/30fa1bf3086a4803ac9e106cf7e47f31
I20260620 14:46:11.629719 4953 ts_tablet_manager.cc:1939] T 30fa1bf3086a4803ac9e106cf7e47f31 P c177a40f10aa4bd58ec5c8448c9541e4: Deleting consensus metadata
I20260620 14:46:11.630115 4445 catalog_manager.cc:4841] TS c177a40f10aa4bd58ec5c8448c9541e4 (127.0.98.3:43977): tablet 30fa1bf3086a4803ac9e106cf7e47f31 (table test-table [id=8a44f1b2187443a6a009c801ada4e917]) successfully deleted
I20260620 14:46:11.631529 4951 tablet_replica.cc:330] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c: stopping tablet replica
I20260620 14:46:11.631642 4951 raft_consensus.cc:2229] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:46:11.631748 4951 raft_consensus.cc:2258] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:11.632314 4951 ts_tablet_manager.cc:1905] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:11.633833 4951 ts_tablet_manager.cc:1918] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:46:11.633909 4951 log.cc:1196] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/30fa1bf3086a4803ac9e106cf7e47f31
I20260620 14:46:11.634166 4951 ts_tablet_manager.cc:1939] T 30fa1bf3086a4803ac9e106cf7e47f31 P 268f71ed200f4ffbb94455bdf94ee03c: Deleting consensus metadata
W20260620 14:46:11.634491 4446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33515 (error 108)
I20260620 14:46:11.634599 4445 catalog_manager.cc:4841] TS 268f71ed200f4ffbb94455bdf94ee03c (127.0.98.2:35195): tablet 30fa1bf3086a4803ac9e106cf7e47f31 (table test-table [id=8a44f1b2187443a6a009c801ada4e917]) successfully deleted
I20260620 14:46:11.634737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 4654
I20260620 14:46:11.635345 4923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.634896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33515 (local address 127.0.0.1:58522)
0620 14:46:11.635265 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:11.635314 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344}
W20260620 14:46:11.635707 4446 catalog_manager.cc:4584] TS 0a160ca8c7914074ba5828309c9b3234 (127.0.98.1:33515): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 30fa1bf3086a4803ac9e106cf7e47f31: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33515: connect: Connection refused (error 111)
I20260620 14:46:11.638852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 4787
I20260620 14:46:11.642849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 4428
2026-06-20T14:46:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1869 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T14:46:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:46:11Z Disabled control of system clock
I20260620 14:46:11.666831 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40571
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:46:11.742369 4958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:11.742564 4958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:11.742583 4958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:11.743654 4958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:46:11.743685 4958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:11.743696 4958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:11.743706 4958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:46:11.743717 4958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:46:11.744958 4958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40571
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:11.745146 4958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:11.745369 4958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:11.749933 4964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:11.750054 4963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:11.749986 4958 server_base.cc:1034] running on GCE node
W20260620 14:46:11.750018 4966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:11.750489 4958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:11.750770 4958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:11.751902 4958 hybrid_clock.cc:648] HybridClock initialized: now 1781966771751884 us; error 30 us; skew 500 ppm
I20260620 14:46:11.753000 4958 webserver.cc:460] Webserver started at http://127.0.98.62:43531/ using document root <none> and password file <none>
I20260620 14:46:11.753228 4958 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:11.753274 4958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:11.753394 4958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:11.754381 4958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "9209178bffc4469685d5ee64e4982d17"
format_stamp: "Formatted at 2026-06-20 14:46:11 on dist-test-slave-cx89"
server_key: "c4efda4d9869662cdd7c052dc409f16e"
server_key_iv: "a8f8dfbd9923abb17202fa4d775660f7"
server_key_version: "encryptionkey@0"
I20260620 14:46:11.754704 4958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "9209178bffc4469685d5ee64e4982d17"
format_stamp: "Formatted at 2026-06-20 14:46:11 on dist-test-slave-cx89"
server_key: "c4efda4d9869662cdd7c052dc409f16e"
server_key_iv: "a8f8dfbd9923abb17202fa4d775660f7"
server_key_version: "encryptionkey@0"
I20260620 14:46:11.755887 4958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:46:11.756520 4972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:11.756659 4958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:46:11.756734 4958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "9209178bffc4469685d5ee64e4982d17"
format_stamp: "Formatted at 2026-06-20 14:46:11 on dist-test-slave-cx89"
server_key: "c4efda4d9869662cdd7c052dc409f16e"
server_key_iv: "a8f8dfbd9923abb17202fa4d775660f7"
server_key_version: "encryptionkey@0"
I20260620 14:46:11.756793 4958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:11.760010 4958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:11.760285 4958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:11.760381 4958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:11.763893 4958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40571
I20260620 14:46:11.763923 5024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40571 every 8 connection(s)
I20260620 14:46:11.764221 4958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:46:11.764779 5025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:11.767311 5025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17: Bootstrap starting.
I20260620 14:46:11.767874 5025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:11.768121 5025 log.cc:824] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:11.768748 5025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17: No bootstrap required, opened a new log
I20260620 14:46:11.770323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 4958
I20260620 14:46:11.770411 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:46:11.770323 5025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9209178bffc4469685d5ee64e4982d17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40571 } }
I20260620 14:46:11.770501 5025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:11.770539 5025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9209178bffc4469685d5ee64e4982d17, State: Initialized, Role: FOLLOWER
I20260620 14:46:11.770584 392 external_mini_cluster.cc:1428] Setting key eec5f067b2434c06f7562f07ee23db44
I20260620 14:46:11.770716 5025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9209178bffc4469685d5ee64e4982d17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40571 } }
I20260620 14:46:11.770804 5025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:46:11.770836 5025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:46:11.770879 5025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:11.771498 5025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9209178bffc4469685d5ee64e4982d17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40571 } }
I20260620 14:46:11.771600 5025 leader_election.cc:302] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9209178bffc4469685d5ee64e4982d17; no voters:
I20260620 14:46:11.771718 5028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:46:11.771799 5028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9209178bffc4469685d5ee64e4982d17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40571 } }
I20260620 14:46:11.772109 5028 leader_election.cc:302] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9209178bffc4469685d5ee64e4982d17; no voters:
I20260620 14:46:11.772109 5025 leader_election.cc:288] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:46:11.772253 5028 leader_election.cc:288] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers
I20260620 14:46:11.772297 5031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:11.772358 5031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [term 1 LEADER]: Becoming Leader. State: Replica: 9209178bffc4469685d5ee64e4982d17, State: Running, Role: LEADER
I20260620 14:46:11.772403 5025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:46:11.772428 5031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9209178bffc4469685d5ee64e4982d17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40571 } }
I20260620 14:46:11.772740 5031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9209178bffc4469685d5ee64e4982d17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9209178bffc4469685d5ee64e4982d17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40571 } } }
I20260620 14:46:11.772746 5028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9209178bffc4469685d5ee64e4982d17. Latest consensus state: current_term: 1 leader_uuid: "9209178bffc4469685d5ee64e4982d17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9209178bffc4469685d5ee64e4982d17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40571 } } }
I20260620 14:46:11.772810 5031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:11.772837 5028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:11.773017 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.771088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48788 (local address 127.0.98.62:40571)
0620 14:46:11.771276 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:11.771284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:11.771307 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:46:11.771403 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:11.771408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:11.771474 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:11.771591 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:11.772329 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:11.772332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:11.772333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:46:11.772337 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:11.772382 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:46:11.772397 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:11.772436 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:11.772572 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 14:46:11.772626 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260620 14:46:11.773197 5040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:46:11.773833 5040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:46:11.775053 5040 catalog_manager.cc:1347] Generated new cluster ID: 96503aa76c00457a808f37f8acde7b50
I20260620 14:46:11.775103 5040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:46:11.795599 5040 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:46:11.795768 5040 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:46:11.802003 5040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17: Generated new TSK 0
I20260620 14:46:11.802170 5040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:46:11.809352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40571
--builtin_ntp_servers=127.0.98.20:44847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:46:11.900178 5050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:11.900369 5050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:11.900389 5050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:11.901515 5050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:11.901551 5050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:11.901575 5050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:46:11.902865 5050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:11.903079 5050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:11.903342 5050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:11.907966 5056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:11.907927 5055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:11.908149 5050 server_base.cc:1034] running on GCE node
W20260620 14:46:11.907922 5058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:11.908471 5050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:11.908715 5050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:11.909869 5050 hybrid_clock.cc:648] HybridClock initialized: now 1781966771909847 us; error 36 us; skew 500 ppm
I20260620 14:46:11.911036 5050 webserver.cc:460] Webserver started at http://127.0.98.1:41647/ using document root <none> and password file <none>
I20260620 14:46:11.911255 5050 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:11.911298 5050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:11.911437 5050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:11.912448 5050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "9947cb9f39094353af4c578ac7076152"
format_stamp: "Formatted at 2026-06-20 14:46:11 on dist-test-slave-cx89"
server_key: "fb52a5c37eeb8b48ef7c766049587582"
server_key_iv: "bb4f9faf0d51a099f744a652e986d306"
server_key_version: "encryptionkey@0"
I20260620 14:46:11.912751 5050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9947cb9f39094353af4c578ac7076152"
format_stamp: "Formatted at 2026-06-20 14:46:11 on dist-test-slave-cx89"
server_key: "fb52a5c37eeb8b48ef7c766049587582"
server_key_iv: "bb4f9faf0d51a099f744a652e986d306"
server_key_version: "encryptionkey@0"
I20260620 14:46:11.914027 5050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:46:11.914721 5064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:11.914880 5050 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:11.914963 5050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "9947cb9f39094353af4c578ac7076152"
format_stamp: "Formatted at 2026-06-20 14:46:11 on dist-test-slave-cx89"
server_key: "fb52a5c37eeb8b48ef7c766049587582"
server_key_iv: "bb4f9faf0d51a099f744a652e986d306"
server_key_version: "encryptionkey@0"
I20260620 14:46:11.915031 5050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:11.918334 5050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:11.918581 5050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:11.918694 5050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:11.918871 5050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:11.919137 5050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:11.919168 5050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:11.919194 5050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:11.919219 5050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:11.924432 5050 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35079
I20260620 14:46:11.924471 5177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35079 every 8 connection(s)
I20260620 14:46:11.924746 5050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:46:11.927320 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.925362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41167 (local address 127.0.98.62:40571)
0620 14:46:11.925407 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:11.925410 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:11.925622 (+ 212us) server_negotiation.cc:407] Connection header received
0620 14:46:11.926066 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:11.926067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:11.926073 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:11.926110 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:11.927157 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:11.927157 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:11.927158 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:46:11.927160 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:11.927172 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:46:11.927174 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:11.927210 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:11.927260 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 14:46:11.927270 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:46:11.927505 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:11.925344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40571 (local address 127.0.98.1:41167)
0620 14:46:11.925515 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:11.925546 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:11.925980 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:11.926161 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:11.926164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:11.926171 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:11.927039 (+ 868us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:46:11.927040 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:11.927066 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:46:11.927093 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:11.927214 (+ 121us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:11.927216 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:11.927218 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:46:11.927261 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 14:46:11.927304 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20260620 14:46:11.927911 5178 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40571
I20260620 14:46:11.927990 5178 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:11.928094 5178 heartbeater.cc:511] Master 127.0.98.62:40571 requested a full tablet report, sending...
I20260620 14:46:11.928512 4989 ts_manager.cc:194] Registered new tserver with Master: 9947cb9f39094353af4c578ac7076152 (127.0.98.1:35079)
I20260620 14:46:11.934267 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 5050
I20260620 14:46:11.934352 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:46:11.934494 392 external_mini_cluster.cc:1428] Setting key d1788fe954c1a162c5565c4a63725fa8
I20260620 14:46:11.935292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40571
--builtin_ntp_servers=127.0.98.20:44847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:46:12.011297 5181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:12.011507 5181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:12.011528 5181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:12.012657 5181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:12.012696 5181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:12.012719 5181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:46:12.014019 5181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:12.014242 5181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:12.014572 5181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:12.019363 5186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:12.019454 5187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:12.019430 5189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:12.019552 5181 server_base.cc:1034] running on GCE node
I20260620 14:46:12.019779 5181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:12.019985 5181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:12.021121 5181 hybrid_clock.cc:648] HybridClock initialized: now 1781966772021101 us; error 32 us; skew 500 ppm
I20260620 14:46:12.022274 5181 webserver.cc:460] Webserver started at http://127.0.98.2:36399/ using document root <none> and password file <none>
I20260620 14:46:12.022495 5181 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:12.022538 5181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:12.022653 5181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:12.023696 5181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "e0287c8801064eb08b074898374c58ef"
format_stamp: "Formatted at 2026-06-20 14:46:12 on dist-test-slave-cx89"
server_key: "e286359208d5d2ef8f4684fedd41b941"
server_key_iv: "6b60fc94a502e87cba4f634c10871e62"
server_key_version: "encryptionkey@0"
I20260620 14:46:12.024004 5181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e0287c8801064eb08b074898374c58ef"
format_stamp: "Formatted at 2026-06-20 14:46:12 on dist-test-slave-cx89"
server_key: "e286359208d5d2ef8f4684fedd41b941"
server_key_iv: "6b60fc94a502e87cba4f634c10871e62"
server_key_version: "encryptionkey@0"
I20260620 14:46:12.025112 5181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:12.025699 5195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:12.025841 5181 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:12.025933 5181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "e0287c8801064eb08b074898374c58ef"
format_stamp: "Formatted at 2026-06-20 14:46:12 on dist-test-slave-cx89"
server_key: "e286359208d5d2ef8f4684fedd41b941"
server_key_iv: "6b60fc94a502e87cba4f634c10871e62"
server_key_version: "encryptionkey@0"
I20260620 14:46:12.025987 5181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:12.029183 5181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:12.029420 5181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:12.029532 5181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:12.029703 5181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:12.029985 5181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:12.030016 5181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:12.030042 5181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:12.030066 5181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:12.035327 5181 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33455
I20260620 14:46:12.035382 5308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33455 every 8 connection(s)
I20260620 14:46:12.035702 5181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:46:12.038555 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.036318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36665 (local address 127.0.98.62:40571)
0620 14:46:12.036360 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:12.036362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:12.036587 (+ 225us) server_negotiation.cc:407] Connection header received
0620 14:46:12.037062 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:12.037063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:12.037071 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:12.037122 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:12.038292 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:12.038294 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:12.038294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:46:12.038300 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:12.038318 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:46:12.038324 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:12.038373 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:12.038471 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 14:46:12.038486 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:46:12.038744 5311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.036308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40571 (local address 127.0.98.2:36665)
0620 14:46:12.036472 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:12.036504 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:12.036966 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:12.037190 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:12.037195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:12.037205 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:12.038127 (+ 922us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:46:12.038128 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:12.038165 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:46:12.038202 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:12.038398 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:12.038404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:12.038406 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:46:12.038480 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:46:12.038542 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260620 14:46:12.039182 5309 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40571
I20260620 14:46:12.039238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 5181
I20260620 14:46:12.039281 5309 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:12.039324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:46:12.039449 5309 heartbeater.cc:511] Master 127.0.98.62:40571 requested a full tablet report, sending...
I20260620 14:46:12.039528 392 external_mini_cluster.cc:1428] Setting key c8ac1fb822fff8c5a56caed4f76b936b
I20260620 14:46:12.040187 4988 ts_manager.cc:194] Registered new tserver with Master: e0287c8801064eb08b074898374c58ef (127.0.98.2:33455)
I20260620 14:46:12.040436 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40571
--builtin_ntp_servers=127.0.98.20:44847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:46:12.118314 5312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:12.118516 5312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:12.118537 5312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:12.119717 5312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:12.119763 5312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:12.119791 5312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:46:12.121085 5312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:12.121265 5312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:12.121503 5312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:12.126111 5318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:12.126166 5317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:12.126127 5320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:12.126658 5312 server_base.cc:1034] running on GCE node
I20260620 14:46:12.126868 5312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:12.127102 5312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:12.128234 5312 hybrid_clock.cc:648] HybridClock initialized: now 1781966772128220 us; error 28 us; skew 500 ppm
I20260620 14:46:12.129472 5312 webserver.cc:460] Webserver started at http://127.0.98.3:36181/ using document root <none> and password file <none>
I20260620 14:46:12.129663 5312 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:12.129702 5312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:12.129793 5312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:12.130769 5312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "950c8adcaabe4b5db53716559207ce22"
format_stamp: "Formatted at 2026-06-20 14:46:12 on dist-test-slave-cx89"
server_key: "742864d2be6e15175f848be570cc96ed"
server_key_iv: "235b0749d3774186602f4e0cae359fd5"
server_key_version: "encryptionkey@0"
I20260620 14:46:12.131057 5312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "950c8adcaabe4b5db53716559207ce22"
format_stamp: "Formatted at 2026-06-20 14:46:12 on dist-test-slave-cx89"
server_key: "742864d2be6e15175f848be570cc96ed"
server_key_iv: "235b0749d3774186602f4e0cae359fd5"
server_key_version: "encryptionkey@0"
I20260620 14:46:12.132251 5312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:46:12.132856 5326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:12.133001 5312 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:46:12.133080 5312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "950c8adcaabe4b5db53716559207ce22"
format_stamp: "Formatted at 2026-06-20 14:46:12 on dist-test-slave-cx89"
server_key: "742864d2be6e15175f848be570cc96ed"
server_key_iv: "235b0749d3774186602f4e0cae359fd5"
server_key_version: "encryptionkey@0"
I20260620 14:46:12.133140 5312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:12.136380 5312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:12.136610 5312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:12.136723 5312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:12.136898 5312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:12.137167 5312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:12.137198 5312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:12.137224 5312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:12.137248 5312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:12.142447 5312 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44611
I20260620 14:46:12.142498 5439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44611 every 8 connection(s)
I20260620 14:46:12.142786 5312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:46:12.144647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 5312
I20260620 14:46:12.144740 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:46:12.144929 392 external_mini_cluster.cc:1428] Setting key 5e024ef894443f3d75aea1cf5ae6bcc7
I20260620 14:46:12.145586 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.143446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49593 (local address 127.0.98.62:40571)
0620 14:46:12.143501 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:12.143504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:12.143702 (+ 198us) server_negotiation.cc:407] Connection header received
0620 14:46:12.144141 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:12.144142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:12.144147 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:12.144194 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:12.145359 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:12.145360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:12.145361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:46:12.145366 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:12.145381 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:46:12.145386 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:12.145436 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:12.145503 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 14:46:12.145519 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:46:12.145763 5442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.143403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40571 (local address 127.0.98.3:49593)
0620 14:46:12.143592 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:12.143625 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:12.144066 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:12.144249 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:12.144252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:12.144260 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:12.145222 (+ 962us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:46:12.145222 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:12.145250 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:46:12.145278 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:12.145438 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:12.145443 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:12.145446 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:46:12.145514 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:46:12.145574 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260620 14:46:12.146169 5440 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40571
I20260620 14:46:12.146248 5440 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:12.146355 5440 heartbeater.cc:511] Master 127.0.98.62:40571 requested a full tablet report, sending...
I20260620 14:46:12.146761 4988 ts_manager.cc:194] Registered new tserver with Master: 950c8adcaabe4b5db53716559207ce22 (127.0.98.3:44611)
I20260620 14:46:12.156335 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:46:12.158691 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.157607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.0.98.62:40571)
0620 14:46:12.157693 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:12.157695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:12.157766 (+ 71us) server_negotiation.cc:407] Connection header received
0620 14:46:12.157863 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:12.157864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:12.157870 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:12.157929 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:12.158452 (+ 523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:12.158453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:12.158454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:46:12.158459 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:12.158476 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:46:12.158481 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:12.158540 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:12.158599 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 14:46:12.158615 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:46:12.160315 4988 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:46:12.161820 4988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:46:12.165563 5453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.163933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.0.98.1:35079)
0620 14:46:12.164104 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:12.164108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:12.164226 (+ 118us) server_negotiation.cc:407] Connection header received
0620 14:46:12.164253 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:12.164255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:12.164262 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:12.164323 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:12.165256 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:12.165265 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:12.165265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:46:12.165268 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:12.165303 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:46:12.165332 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:12.165403 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:12.165467 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 14:46:12.165482 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
I20260620 14:46:12.165567 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.163706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35079 (local address 127.0.0.1:37764)
0620 14:46:12.164134 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:12.164141 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:12.164184 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:12.164593 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:12.164594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:12.164600 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:12.165127 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:46:12.165128 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:12.165146 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:46:12.165169 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:12.165389 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:12.165395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:12.165398 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:46:12.165460 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:46:12.165477 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":12,"thread_start_us":84,"threads_started":1}
I20260620 14:46:12.165619 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.163706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33455 (local address 127.0.0.1:41346)
0620 14:46:12.164037 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:12.164045 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:12.164076 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:12.164590 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:12.164591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:12.164594 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:12.165234 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:46:12.165235 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:12.165239 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:46:12.165248 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:12.165459 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:12.165460 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:12.165461 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:46:12.165501 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 14:46:12.165512 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":80,"threads_started":1}
I20260620 14:46:12.165575 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.163696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44611 (local address 127.0.0.1:38828)
0620 14:46:12.163877 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:12.163914 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:12.163966 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:12.164288 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:12.164293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:12.164299 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:12.165166 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:46:12.165166 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:12.165172 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:46:12.165182 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:12.165413 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:12.165414 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:12.165414 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 14:46:12.165462 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:46:12.165475 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":58,"threads_started":1}
I20260620 14:46:12.165800 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.163771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38828 (local address 127.0.98.3:44611)
0620 14:46:12.164167 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:12.164171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:12.164180 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:46:12.164204 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:12.164206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:12.164214 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:12.164291 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:12.165264 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:12.165273 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:12.165274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:46:12.165278 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:12.165315 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:46:12.165343 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:12.165392 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:12.165702 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 14:46:12.165719 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":81,"threads_started":1}
I20260620 14:46:12.166143 5452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.163923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41346 (local address 127.0.98.2:33455)
0620 14:46:12.164098 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:12.164103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:12.164113 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:46:12.164139 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:12.164141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:12.164153 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:12.164222 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:12.165321 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:12.165327 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:12.165328 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:46:12.165331 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:12.165361 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:46:12.165384 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:12.165435 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:12.165600 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 14:46:12.166068 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20260620 14:46:12.166301 5112 tablet_service.cc:1467] Processing CreateTablet for tablet 67be33ab26af4fd4b0d0b2749441cc4b (DEFAULT_TABLE table=test-table [id=205941b1b473443e9c06097b7cdccfaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:12.166404 5374 tablet_service.cc:1467] Processing CreateTablet for tablet 67be33ab26af4fd4b0d0b2749441cc4b (DEFAULT_TABLE table=test-table [id=205941b1b473443e9c06097b7cdccfaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:12.166669 5112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67be33ab26af4fd4b0d0b2749441cc4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:12.166718 5374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67be33ab26af4fd4b0d0b2749441cc4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:12.166718 5243 tablet_service.cc:1467] Processing CreateTablet for tablet 67be33ab26af4fd4b0d0b2749441cc4b (DEFAULT_TABLE table=test-table [id=205941b1b473443e9c06097b7cdccfaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:12.166985 5243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67be33ab26af4fd4b0d0b2749441cc4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:12.169123 5454 tablet_bootstrap.cc:492] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152: Bootstrap starting.
I20260620 14:46:12.169639 5454 tablet_bootstrap.cc:654] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:12.169845 5455 tablet_bootstrap.cc:492] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22: Bootstrap starting.
I20260620 14:46:12.169943 5454 log.cc:824] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:12.170158 5456 tablet_bootstrap.cc:492] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef: Bootstrap starting.
I20260620 14:46:12.170439 5455 tablet_bootstrap.cc:654] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:12.170715 5454 tablet_bootstrap.cc:492] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152: No bootstrap required, opened a new log
I20260620 14:46:12.170723 5456 tablet_bootstrap.cc:654] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:12.170717 5455 log.cc:824] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:12.170779 5454 ts_tablet_manager.cc:1397] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:46:12.171017 5456 log.cc:824] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:12.171351 5455 tablet_bootstrap.cc:492] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22: No bootstrap required, opened a new log
I20260620 14:46:12.171432 5455 ts_tablet_manager.cc:1397] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:46:12.171794 5456 tablet_bootstrap.cc:492] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef: No bootstrap required, opened a new log
I20260620 14:46:12.171900 5456 ts_tablet_manager.cc:1397] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:46:12.172931 5454 raft_consensus.cc:348] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950c8adcaabe4b5db53716559207ce22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44611 } } peers { permanent_uuid: "e0287c8801064eb08b074898374c58ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33455 } } peers { permanent_uuid: "9947cb9f39094353af4c578ac7076152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35079 } }
I20260620 14:46:12.173061 5454 raft_consensus.cc:374] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:12.173090 5454 raft_consensus.cc:729] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9947cb9f39094353af4c578ac7076152, State: Initialized, Role: FOLLOWER
I20260620 14:46:12.173218 5454 consensus_queue.cc:260] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950c8adcaabe4b5db53716559207ce22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44611 } } peers { permanent_uuid: "e0287c8801064eb08b074898374c58ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33455 } } peers { permanent_uuid: "9947cb9f39094353af4c578ac7076152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35079 } }
I20260620 14:46:12.173492 5454 ts_tablet_manager.cc:1428] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:46:12.173617 5178 heartbeater.cc:503] Master 127.0.98.62:40571 was elected leader, sending a full tablet report...
I20260620 14:46:12.173820 5455 raft_consensus.cc:348] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950c8adcaabe4b5db53716559207ce22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44611 } } peers { permanent_uuid: "e0287c8801064eb08b074898374c58ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33455 } } peers { permanent_uuid: "9947cb9f39094353af4c578ac7076152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35079 } }
I20260620 14:46:12.174001 5455 raft_consensus.cc:374] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:12.174039 5455 raft_consensus.cc:729] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 950c8adcaabe4b5db53716559207ce22, State: Initialized, Role: FOLLOWER
I20260620 14:46:12.174142 5455 consensus_queue.cc:260] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950c8adcaabe4b5db53716559207ce22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44611 } } peers { permanent_uuid: "e0287c8801064eb08b074898374c58ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33455 } } peers { permanent_uuid: "9947cb9f39094353af4c578ac7076152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35079 } }
I20260620 14:46:12.174387 5455 ts_tablet_manager.cc:1428] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:46:12.174500 5440 heartbeater.cc:503] Master 127.0.98.62:40571 was elected leader, sending a full tablet report...
I20260620 14:46:12.174536 5456 raft_consensus.cc:348] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950c8adcaabe4b5db53716559207ce22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44611 } } peers { permanent_uuid: "e0287c8801064eb08b074898374c58ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33455 } } peers { permanent_uuid: "9947cb9f39094353af4c578ac7076152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35079 } }
I20260620 14:46:12.174686 5456 raft_consensus.cc:374] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:12.174723 5456 raft_consensus.cc:729] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0287c8801064eb08b074898374c58ef, State: Initialized, Role: FOLLOWER
I20260620 14:46:12.174818 5456 consensus_queue.cc:260] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950c8adcaabe4b5db53716559207ce22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44611 } } peers { permanent_uuid: "e0287c8801064eb08b074898374c58ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33455 } } peers { permanent_uuid: "9947cb9f39094353af4c578ac7076152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35079 } }
I20260620 14:46:12.175063 5309 heartbeater.cc:503] Master 127.0.98.62:40571 was elected leader, sending a full tablet report...
W20260620 14:46:12.175151 5179 tablet.cc:2369] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:46:12.175060 5456 ts_tablet_manager.cc:1428] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:46:12.278589 5460 raft_consensus.cc:482] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:46:12.278704 5460 raft_consensus.cc:504] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950c8adcaabe4b5db53716559207ce22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44611 } } peers { permanent_uuid: "e0287c8801064eb08b074898374c58ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33455 } } peers { permanent_uuid: "9947cb9f39094353af4c578ac7076152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35079 } }
I20260620 14:46:12.279053 5460 leader_election.cc:288] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 950c8adcaabe4b5db53716559207ce22 (127.0.98.3:44611), e0287c8801064eb08b074898374c58ef (127.0.98.2:33455)
I20260620 14:46:12.280309 5452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.279309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38603 (local address 127.0.98.2:33455)
0620 14:46:12.279362 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:12.279364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:12.279376 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:46:12.279402 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:12.279404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:12.279409 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:12.279477 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:12.280087 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:12.280088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:12.280089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:46:12.280093 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:12.280112 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:46:12.280117 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:12.280177 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:12.280223 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 14:46:12.280238 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:46:12.280314 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.279274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34425 (local address 127.0.98.3:44611)
0620 14:46:12.279312 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:12.279313 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:12.279438 (+ 125us) server_negotiation.cc:407] Connection header received
0620 14:46:12.279506 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:12.279507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:12.279511 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:12.279555 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:12.280114 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:12.280116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:12.280116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:46:12.280120 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:12.280136 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:46:12.280140 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:12.280191 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:12.280245 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 14:46:12.280257 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:46:12.280311 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.279212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33455 (local address 127.0.98.1:38603)
0620 14:46:12.279280 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:12.279293 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:12.279343 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:12.279536 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:12.279537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:12.279539 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:12.279987 (+ 448us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:46:12.279988 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:12.279996 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:46:12.280010 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:12.280175 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:12.280176 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:12.280177 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:46:12.280224 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:46:12.280236 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 14:46:12.280323 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.279212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44611 (local address 127.0.98.1:34425)
0620 14:46:12.279362 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:12.279378 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:12.279453 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:12.279578 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:12.279578 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:12.279580 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:12.280032 (+ 452us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:46:12.280033 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:12.280037 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:46:12.280045 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:12.280191 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:12.280192 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:12.280192 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 14:46:12.280242 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:46:12.280251 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":13,"thread_start_us":80,"threads_started":1}
I20260620 14:46:12.280622 5394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67be33ab26af4fd4b0d0b2749441cc4b" candidate_uuid: "9947cb9f39094353af4c578ac7076152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "950c8adcaabe4b5db53716559207ce22" is_pre_election: true
I20260620 14:46:12.280680 5263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67be33ab26af4fd4b0d0b2749441cc4b" candidate_uuid: "9947cb9f39094353af4c578ac7076152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0287c8801064eb08b074898374c58ef" is_pre_election: true
I20260620 14:46:12.280754 5394 raft_consensus.cc:2454] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9947cb9f39094353af4c578ac7076152 in term 0.
I20260620 14:46:12.280825 5263 raft_consensus.cc:2454] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9947cb9f39094353af4c578ac7076152 in term 0.
I20260620 14:46:12.281001 5066 leader_election.cc:302] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 950c8adcaabe4b5db53716559207ce22, 9947cb9f39094353af4c578ac7076152; no voters:
I20260620 14:46:12.281134 5460 raft_consensus.cc:2781] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:46:12.281193 5460 raft_consensus.cc:482] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:46:12.281209 5460 raft_consensus.cc:3037] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:12.281872 5460 raft_consensus.cc:504] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950c8adcaabe4b5db53716559207ce22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44611 } } peers { permanent_uuid: "e0287c8801064eb08b074898374c58ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33455 } } peers { permanent_uuid: "9947cb9f39094353af4c578ac7076152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35079 } }
I20260620 14:46:12.281987 5460 leader_election.cc:288] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 [CANDIDATE]: Term 1 election: Requested vote from peers 950c8adcaabe4b5db53716559207ce22 (127.0.98.3:44611), e0287c8801064eb08b074898374c58ef (127.0.98.2:33455)
I20260620 14:46:12.282114 5263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67be33ab26af4fd4b0d0b2749441cc4b" candidate_uuid: "9947cb9f39094353af4c578ac7076152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0287c8801064eb08b074898374c58ef"
I20260620 14:46:12.282171 5394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67be33ab26af4fd4b0d0b2749441cc4b" candidate_uuid: "9947cb9f39094353af4c578ac7076152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "950c8adcaabe4b5db53716559207ce22"
I20260620 14:46:12.282192 5263 raft_consensus.cc:3037] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:12.282243 5394 raft_consensus.cc:3037] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:12.283039 5263 raft_consensus.cc:2454] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9947cb9f39094353af4c578ac7076152 in term 1.
I20260620 14:46:12.283039 5394 raft_consensus.cc:2454] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9947cb9f39094353af4c578ac7076152 in term 1.
I20260620 14:46:12.283238 5068 leader_election.cc:302] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9947cb9f39094353af4c578ac7076152, e0287c8801064eb08b074898374c58ef; no voters:
I20260620 14:46:12.283321 5460 raft_consensus.cc:2781] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:12.283500 5460 raft_consensus.cc:686] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 [term 1 LEADER]: Becoming Leader. State: Replica: 9947cb9f39094353af4c578ac7076152, State: Running, Role: LEADER
I20260620 14:46:12.283593 5460 consensus_queue.cc:237] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950c8adcaabe4b5db53716559207ce22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44611 } } peers { permanent_uuid: "e0287c8801064eb08b074898374c58ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33455 } } peers { permanent_uuid: "9947cb9f39094353af4c578ac7076152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35079 } }
I20260620 14:46:12.284379 4989 catalog_manager.cc:5495] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9947cb9f39094353af4c578ac7076152 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9947cb9f39094353af4c578ac7076152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "950c8adcaabe4b5db53716559207ce22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0287c8801064eb08b074898374c58ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9947cb9f39094353af4c578ac7076152" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35079 } health_report { overall_health: HEALTHY } } }
W20260620 14:46:12.286165 5310 tablet.cc:2369] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:46:12.309131 5453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.308236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37780 (local address 127.0.98.1:35079)
0620 14:46:12.308288 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:12.308290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:12.308298 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:46:12.308336 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:12.308338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:12.308352 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:12.308397 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:12.308920 (+ 523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:12.308921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:12.308922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:46:12.308925 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:12.308942 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:46:12.308946 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:12.308993 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:12.309045 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 14:46:12.309057 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:46:12.310765 5394 raft_consensus.cc:1261] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22 [term 1 FOLLOWER]: Refusing update from remote peer 9947cb9f39094353af4c578ac7076152: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:12.310772 5263 raft_consensus.cc:1261] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef [term 1 FOLLOWER]: Refusing update from remote peer 9947cb9f39094353af4c578ac7076152: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:12.311077 5460 consensus_queue.cc:1035] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "950c8adcaabe4b5db53716559207ce22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:46:12.311211 5464 consensus_queue.cc:1035] T 67be33ab26af4fd4b0d0b2749441cc4b P 9947cb9f39094353af4c578ac7076152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0287c8801064eb08b074898374c58ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:46:12.314138 5468 mvcc.cc:205] Tried to move back new op lower bound from 7298935899382235136 to 7298935899274051584. Current Snapshot: MvccSnapshot[applied={T|T < 7298935899382235136}]
I20260620 14:46:12.315459 4989 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:12.315549 4989 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:12.316215 4989 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9209178bffc4469685d5ee64e4982d17: Sending DeleteTablet for 3 replicas of tablet 67be33ab26af4fd4b0d0b2749441cc4b
I20260620 14:46:12.316529 5243 tablet_service.cc:1514] Processing DeleteTablet for tablet 67be33ab26af4fd4b0d0b2749441cc4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:12 UTC) from {username='slave'} at 127.0.0.1:41346
I20260620 14:46:12.316522 5374 tablet_service.cc:1514] Processing DeleteTablet for tablet 67be33ab26af4fd4b0d0b2749441cc4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:12 UTC) from {username='slave'} at 127.0.0.1:38828
I20260620 14:46:12.316814 5477 tablet_replica.cc:330] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef: stopping tablet replica
I20260620 14:46:12.316846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 5050
I20260620 14:46:12.316830 5112 tablet_service.cc:1514] Processing DeleteTablet for tablet 67be33ab26af4fd4b0d0b2749441cc4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:12 UTC) from {username='slave'} at 127.0.0.1:37764
I20260620 14:46:12.316902 5477 raft_consensus.cc:2229] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:46:12.317004 5477 raft_consensus.cc:2258] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:12.317343 5476 tablet_replica.cc:330] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22: stopping tablet replica
I20260620 14:46:12.317427 5476 raft_consensus.cc:2229] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:46:12.317513 5476 raft_consensus.cc:2258] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:12.317580 5477 ts_tablet_manager.cc:1905] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:12.317889 5476 ts_tablet_manager.cc:1905] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:12.319559 5476 ts_tablet_manager.cc:1918] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:46:12.319598 5477 ts_tablet_manager.cc:1918] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:46:12.319621 5476 log.cc:1196] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/67be33ab26af4fd4b0d0b2749441cc4b
I20260620 14:46:12.319667 5477 log.cc:1196] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/67be33ab26af4fd4b0d0b2749441cc4b
I20260620 14:46:12.319862 5476 ts_tablet_manager.cc:1939] T 67be33ab26af4fd4b0d0b2749441cc4b P 950c8adcaabe4b5db53716559207ce22: Deleting consensus metadata
I20260620 14:46:12.319877 5477 ts_tablet_manager.cc:1939] T 67be33ab26af4fd4b0d0b2749441cc4b P e0287c8801064eb08b074898374c58ef: Deleting consensus metadata
I20260620 14:46:12.320200 4976 catalog_manager.cc:4841] TS e0287c8801064eb08b074898374c58ef (127.0.98.2:33455): tablet 67be33ab26af4fd4b0d0b2749441cc4b (table test-table [id=205941b1b473443e9c06097b7cdccfaf]) successfully deleted
I20260620 14:46:12.320305 4974 catalog_manager.cc:4841] TS 950c8adcaabe4b5db53716559207ce22 (127.0.98.3:44611): tablet 67be33ab26af4fd4b0d0b2749441cc4b (table test-table [id=205941b1b473443e9c06097b7cdccfaf]) successfully deleted
W20260620 14:46:12.322687 4975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35079 (error 108)
I20260620 14:46:12.322901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 5181
I20260620 14:46:12.323395 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.322988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35079 (local address 127.0.0.1:37786)
0620 14:46:12.323322 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:12.323362 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314}
W20260620 14:46:12.323652 4975 catalog_manager.cc:4584] TS 9947cb9f39094353af4c578ac7076152 (127.0.98.1:35079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67be33ab26af4fd4b0d0b2749441cc4b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35079: connect: Connection refused (error 111)
I20260620 14:46:12.327365 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 5312
I20260620 14:46:12.330976 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 4958
2026-06-20T14:46:12Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (687 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 14:46:12.348587 392 mini_oidc.cc:152] Starting JWKS server
I20260620 14:46:12.349581 392 webserver.cc:460] Webserver started at https://127.0.0.1:36007/ using document root <none> and password file <none>
I20260620 14:46:12.349660 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:46:12.349812 392 webserver.cc:460] Webserver started at http://127.0.0.1:45991/ using document root <none> and password file <none>
2026-06-20T14:46:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:46:12Z Disabled control of system clock
I20260620 14:46:12.356487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43629
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43629
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36007/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0/testchainca.cert with env {}
W20260620 14:46:12.433197 5484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:12.433398 5484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:12.433418 5484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:12.434547 5484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:46:12.434593 5484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:12.434605 5484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:12.434616 5484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:46:12.434626 5484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:46:12.434636 5484 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:46:12.434646 5484 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36007/jwks.json
I20260620 14:46:12.436157 5484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43629
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36007/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:12.436349 5484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:12.436519 5484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:12.440972 5489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:12.441064 5484 server_base.cc:1034] running on GCE node
W20260620 14:46:12.440959 5490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:12.440963 5492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:12.441417 5484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:12.441702 5484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:12.442860 5484 hybrid_clock.cc:648] HybridClock initialized: now 1781966772442844 us; error 31 us; skew 500 ppm
I20260620 14:46:12.444208 5484 webserver.cc:460] Webserver started at http://127.0.98.62:40293/ using document root <none> and password file <none>
I20260620 14:46:12.444450 5484 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:12.444501 5484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:12.444610 5484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:12.445655 5484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "45299d1583dd470a83649d54e90ef42c"
format_stamp: "Formatted at 2026-06-20 14:46:12 on dist-test-slave-cx89"
server_key: "4ecea60244aabaeddffb44c7c650810c"
server_key_iv: "38d646c1022189fc6d515e5bb83f6c1f"
server_key_version: "encryptionkey@0"
I20260620 14:46:12.445992 5484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "45299d1583dd470a83649d54e90ef42c"
format_stamp: "Formatted at 2026-06-20 14:46:12 on dist-test-slave-cx89"
server_key: "4ecea60244aabaeddffb44c7c650810c"
server_key_iv: "38d646c1022189fc6d515e5bb83f6c1f"
server_key_version: "encryptionkey@0"
I20260620 14:46:12.447285 5484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:46:12.448016 5498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:12.448158 5484 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:12.448238 5484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "45299d1583dd470a83649d54e90ef42c"
format_stamp: "Formatted at 2026-06-20 14:46:12 on dist-test-slave-cx89"
server_key: "4ecea60244aabaeddffb44c7c650810c"
server_key_iv: "38d646c1022189fc6d515e5bb83f6c1f"
server_key_version: "encryptionkey@0"
I20260620 14:46:12.448297 5484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:12.460529 5484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:12.460815 5484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:12.460942 5484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:12.464524 5484 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43629
I20260620 14:46:12.464557 5550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43629 every 8 connection(s)
I20260620 14:46:12.464860 5484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:46:12.465709 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:12.467880 5551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c: Bootstrap starting.
I20260620 14:46:12.468391 5551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:12.468612 5551 log.cc:824] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:12.469170 5551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c: No bootstrap required, opened a new log
I20260620 14:46:12.470125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 5484
I20260620 14:46:12.470232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:46:12.470408 392 external_mini_cluster.cc:1428] Setting key 64e48c286e8090c7f5d16eedec7aab26
I20260620 14:46:12.470976 5551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45299d1583dd470a83649d54e90ef42c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } }
I20260620 14:46:12.471148 5551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:12.471184 5551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45299d1583dd470a83649d54e90ef42c, State: Initialized, Role: FOLLOWER
I20260620 14:46:12.471303 5551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45299d1583dd470a83649d54e90ef42c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } }
I20260620 14:46:12.471371 5551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:46:12.471403 5551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:46:12.471477 5551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:12.472415 5551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45299d1583dd470a83649d54e90ef42c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } }
I20260620 14:46:12.472527 5551 leader_election.cc:302] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45299d1583dd470a83649d54e90ef42c; no voters:
I20260620 14:46:12.472669 5551 leader_election.cc:288] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:46:12.472715 5556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:12.472862 5556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c [term 1 LEADER]: Becoming Leader. State: Replica: 45299d1583dd470a83649d54e90ef42c, State: Running, Role: LEADER
I20260620 14:46:12.472898 5551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:46:12.472949 5556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45299d1583dd470a83649d54e90ef42c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } }
I20260620 14:46:12.473271 5556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45299d1583dd470a83649d54e90ef42c. Latest consensus state: current_term: 1 leader_uuid: "45299d1583dd470a83649d54e90ef42c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45299d1583dd470a83649d54e90ef42c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } }
I20260620 14:46:12.473340 5556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:12.473482 5557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45299d1583dd470a83649d54e90ef42c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45299d1583dd470a83649d54e90ef42c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } }
I20260620 14:46:12.473526 5557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:12.473989 5563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:46:12.474495 5563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:46:12.475965 5563 catalog_manager.cc:1347] Generated new cluster ID: cc0da0fa3ae3482d9bdb9f9efa3570f8
I20260620 14:46:12.476014 5563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:46:12.476279 5555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.470908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.0.98.62:43629)
0620 14:46:12.471146 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:12.471154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:12.471184 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:46:12.471293 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:12.471299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:12.471368 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:12.471577 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:12.471760 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:12.473111 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:12.474212 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:12.474807 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:12.474926 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:12.475659 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:12.475663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:12.475664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:46:12.475826 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:12.475870 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:46:12.475885 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:12.475940 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:12.476017 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 14:46:12.476090 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":80,"threads_started":1}
I20260620 14:46:12.477061 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:46:12.482925 5563 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:46:12.483636 5563 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:46:12.490170 5563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45299d1583dd470a83649d54e90ef42c: Generated new TSK 0
I20260620 14:46:12.490358 5563 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:46:12.515493 5581 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 14:46:12.515599 5581 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:46:12.515808 5555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.501017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32796 (local address 127.0.98.62:43629)
0620 14:46:12.501069 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:12.501071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:12.501164 (+ 93us) server_negotiation.cc:407] Connection header received
0620 14:46:12.501261 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:12.501262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:12.501272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:12.501326 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:46:12.501449 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:12.501921 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:12.503036 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:12.503220 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:12.503265 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:12.503500 (+ 235us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:46:12.515592 (+ 12092us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:46:12.515745 (+ 153us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32796: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 14:46:12.515873 5555 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32796: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 14:46:12.516142 5580 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:12.500926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43629 (local address 127.0.0.1:32796)
0620 14:46:12.501093 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:12.501104 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:12.501178 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:12.501333 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:12.501333 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:12.501335 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:46:12.501399 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:12.501401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:12.501997 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:12.501998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:12.502951 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:12.502953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:12.503278 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:12.503281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:12.503344 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:12.503366 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:46:12.515766 (+ 12400us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:46:12.515867 (+ 101us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43629: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:43629: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 14:46:12.516516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 5484
2026-06-20T14:46:12Z chronyd exiting
I20260620 14:46:12.552019 392 test_util.cc:175] -----------------------------------------------
I20260620 14:46:12.552096 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966748446763-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (204 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 14:46:12.552824 392 mini_oidc.cc:152] Starting JWKS server
I20260620 14:46:12.553483 392 webserver.cc:460] Webserver started at https://127.0.0.1:38425/ using document root <none> and password file <none>
I20260620 14:46:12.553526 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:46:12.553663 392 webserver.cc:460] Webserver started at http://127.0.0.1:44877/ using document root <none> and password file <none>
2026-06-20T14:46:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:46:12Z Disabled control of system clock
I20260620 14:46:12.560297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33681
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33681
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38425/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966748446763-392-0/testchainca.cert with env {}
W20260620 14:46:12.637231 5587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:12.637439 5587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:12.637468 5587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:12.638598 5587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:46:12.638638 5587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:12.638654 5587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:12.638669 5587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:46:12.638684 5587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:46:12.638707 5587 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:46:12.638723 5587 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38425/jwks.json
I20260620 14:46:12.640249 5587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33681
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38425/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966748446763-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:12.640492 5587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:12.640687 5587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:12.645262 5593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:12.645298 5587 server_base.cc:1034] running on GCE node
W20260620 14:46:12.645367 5592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:12.645367 5595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:12.645695 5587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:12.646000 5587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:12.647148 5587 hybrid_clock.cc:648] HybridClock initialized: now 1781966772647130 us; error 33 us; skew 500 ppm
I20260620 14:46:12.648350 5587 webserver.cc:460] Webserver started at http://127.0.98.62:34607/ using document root <none> and password file <none>
I20260620 14:46:12.648563 5587 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:12.648609 5587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:12.648729 5587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:12.649741 5587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "764ef7faaf484b1bafe01629c11c4561"
format_stamp: "Formatted at 2026-06-20 14:46:12 on dist-test-slave-cx89"
server_key: "a754800d07dd16b15f903495e2546917"
server_key_iv: "98919553f11fc635d46f1800278b81f3"
server_key_version: "encryptionkey@0"
I20260620 14:46:12.650071 5587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "764ef7faaf484b1bafe01629c11c4561"
format_stamp: "Formatted at 2026-06-20 14:46:12 on dist-test-slave-cx89"
server_key: "a754800d07dd16b15f903495e2546917"
server_key_iv: "98919553f11fc635d46f1800278b81f3"
server_key_version: "encryptionkey@0"
I20260620 14:46:12.651224 5587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:46:12.651901 5601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:12.652045 5587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:12.652123 5587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "764ef7faaf484b1bafe01629c11c4561"
format_stamp: "Formatted at 2026-06-20 14:46:12 on dist-test-slave-cx89"
server_key: "a754800d07dd16b15f903495e2546917"
server_key_iv: "98919553f11fc635d46f1800278b81f3"
server_key_version: "encryptionkey@0"
I20260620 14:46:12.652187 5587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:12.659860 5587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:12.660089 5587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:12.660202 5587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:12.663635 5587 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33681
I20260620 14:46:12.663662 5653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33681 every 8 connection(s)
I20260620 14:46:12.663975 5587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:46:12.664834 5654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:12.666898 5654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561: Bootstrap starting.
I20260620 14:46:12.667403 5654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:12.667678 5654 log.cc:824] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:12.668246 5654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561: No bootstrap required, opened a new log
I20260620 14:46:12.669714 5654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764ef7faaf484b1bafe01629c11c4561" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33681 } }
I20260620 14:46:12.669842 5654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:12.669863 5654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 764ef7faaf484b1bafe01629c11c4561, State: Initialized, Role: FOLLOWER
I20260620 14:46:12.669996 5654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764ef7faaf484b1bafe01629c11c4561" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33681 } }
I20260620 14:46:12.670053 5654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:46:12.670071 5654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:46:12.670096 5654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:12.670557 5654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764ef7faaf484b1bafe01629c11c4561" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33681 } }
I20260620 14:46:12.670637 5654 leader_election.cc:302] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 764ef7faaf484b1bafe01629c11c4561; no voters:
I20260620 14:46:12.670774 5654 leader_election.cc:288] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:46:12.670814 5657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:12.670972 5657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561 [term 1 LEADER]: Becoming Leader. State: Replica: 764ef7faaf484b1bafe01629c11c4561, State: Running, Role: LEADER
I20260620 14:46:12.671000 5654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:46:12.671072 5657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764ef7faaf484b1bafe01629c11c4561" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33681 } }
I20260620 14:46:12.671440 5658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "764ef7faaf484b1bafe01629c11c4561" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764ef7faaf484b1bafe01629c11c4561" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33681 } } }
I20260620 14:46:12.671533 5659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 764ef7faaf484b1bafe01629c11c4561. Latest consensus state: current_term: 1 leader_uuid: "764ef7faaf484b1bafe01629c11c4561" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764ef7faaf484b1bafe01629c11c4561" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33681 } } }
I20260620 14:46:12.671594 5659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:12.671554 5658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:12.672047 5666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:46:12.672518 5666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:46:12.673884 5666 catalog_manager.cc:1347] Generated new cluster ID: 01f6452fa04a469bae6df8fe566416cb
I20260620 14:46:12.673935 5666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:46:12.674014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 5587
I20260620 14:46:12.674111 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:46:12.674283 392 external_mini_cluster.cc:1428] Setting key 8d7eaa272df73c9b75ba1ebfc87e433d
I20260620 14:46:12.677937 5666 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:46:12.678583 5666 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:46:12.680908 5677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.675516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40658 (local address 127.0.98.62:33681)
0620 14:46:12.675710 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:12.675719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:12.675756 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:46:12.675869 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:12.675876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:12.675942 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:12.676089 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:12.676298 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:12.677679 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:12.678729 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:12.679380 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:12.679544 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:12.680211 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:12.680214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:12.680215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:46:12.680414 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:12.680455 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:46:12.680469 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:12.680537 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:12.680599 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 14:46:12.680688 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260620 14:46:12.681893 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:46:12.688640 5666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 764ef7faaf484b1bafe01629c11c4561: Generated new TSK 0
I20260620 14:46:12.688805 5666 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:46:12.710402 5684 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 14:46:12.710492 5684 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:46:12.710757 5677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.696830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40660 (local address 127.0.98.62:33681)
0620 14:46:12.696869 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:12.696871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:12.696913 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:46:12.697001 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:12.697002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:12.697011 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:12.697066 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:46:12.697189 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:12.697681 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:12.698760 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:12.698965 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:12.699014 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:12.699165 (+ 151us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:46:12.710582 (+ 11417us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:46:12.710707 (+ 125us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40660: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 14:46:12.710800 5677 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40660: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 14:46:12.710947 5683 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:12.696726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33681 (local address 127.0.0.1:40660)
0620 14:46:12.696857 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:12.696867 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:12.696930 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:12.697075 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:12.697076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:12.697078 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:46:12.697135 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:12.697137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:12.697738 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:12.697739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:12.698682 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:12.698684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:12.699025 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:12.699028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:12.699084 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:12.699090 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:46:12.710717 (+ 11627us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:46:12.710845 (+ 128us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33681: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260620 14:46:12.711208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 5587
2026-06-20T14:46:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (194 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 14:46:12.747635 392 mini_oidc.cc:152] Starting JWKS server
I20260620 14:46:12.748296 392 webserver.cc:460] Webserver started at https://127.0.0.1:44895/ using document root <none> and password file <none>
I20260620 14:46:12.748342 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:46:12.748479 392 webserver.cc:460] Webserver started at http://127.0.0.1:46713/ using document root <none> and password file <none>
2026-06-20T14:46:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:46:12Z Disabled control of system clock
I20260620 14:46:12.755482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38649
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38649
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44895/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:46:12.836534 5690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:12.836732 5690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:12.836752 5690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:12.837895 5690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:46:12.837934 5690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:12.837946 5690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:12.837957 5690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:46:12.837980 5690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:46:12.838002 5690 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:46:12.838022 5690 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44895/jwks.json
I20260620 14:46:12.839375 5690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38649
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44895/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:12.839617 5690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:12.839896 5690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:12.844463 5698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:12.844419 5696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:12.844465 5690 server_base.cc:1034] running on GCE node
W20260620 14:46:12.844432 5695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:12.844842 5690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:12.845114 5690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:12.846243 5690 hybrid_clock.cc:648] HybridClock initialized: now 1781966772846232 us; error 29 us; skew 500 ppm
I20260620 14:46:12.847347 5690 webserver.cc:460] Webserver started at http://127.0.98.62:45467/ using document root <none> and password file <none>
I20260620 14:46:12.847582 5690 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:12.847627 5690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:12.847740 5690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:12.848750 5690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "cd9b761b00c443e7a782eebc0bfc6b01"
format_stamp: "Formatted at 2026-06-20 14:46:12 on dist-test-slave-cx89"
server_key: "1f6b0934245190157ace036f2ad64494"
server_key_iv: "e2084a15bdacfe7459f8140a0d9f57a3"
server_key_version: "encryptionkey@0"
I20260620 14:46:12.849077 5690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "cd9b761b00c443e7a782eebc0bfc6b01"
format_stamp: "Formatted at 2026-06-20 14:46:12 on dist-test-slave-cx89"
server_key: "1f6b0934245190157ace036f2ad64494"
server_key_iv: "e2084a15bdacfe7459f8140a0d9f57a3"
server_key_version: "encryptionkey@0"
I20260620 14:46:12.850270 5690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:46:12.850965 5704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:12.851135 5690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:12.851217 5690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "cd9b761b00c443e7a782eebc0bfc6b01"
format_stamp: "Formatted at 2026-06-20 14:46:12 on dist-test-slave-cx89"
server_key: "1f6b0934245190157ace036f2ad64494"
server_key_iv: "e2084a15bdacfe7459f8140a0d9f57a3"
server_key_version: "encryptionkey@0"
I20260620 14:46:12.851269 5690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:12.863193 5690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:12.863497 5690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:12.863603 5690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:12.866928 5690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38649
I20260620 14:46:12.866991 5756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38649 every 8 connection(s)
I20260620 14:46:12.867273 5690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:46:12.867918 5757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:12.869449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 5690
I20260620 14:46:12.869551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:46:12.869747 392 external_mini_cluster.cc:1428] Setting key 3541231e0e7bba3f50e4294500fc6ebe
I20260620 14:46:12.870850 5757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01: Bootstrap starting.
I20260620 14:46:12.871440 5757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:12.871712 5757 log.cc:824] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:12.872427 5757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01: No bootstrap required, opened a new log
I20260620 14:46:12.874132 5757 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9b761b00c443e7a782eebc0bfc6b01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38649 } }
I20260620 14:46:12.874290 5757 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:12.874329 5757 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd9b761b00c443e7a782eebc0bfc6b01, State: Initialized, Role: FOLLOWER
I20260620 14:46:12.874462 5757 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9b761b00c443e7a782eebc0bfc6b01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38649 } }
I20260620 14:46:12.874536 5757 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:46:12.874573 5757 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:46:12.874634 5757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:12.875036 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.870268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45670 (local address 127.0.98.62:38649)
0620 14:46:12.870455 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:12.870462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:12.870494 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:46:12.870879 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:12.870885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:12.870961 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:12.871091 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:12.871269 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:12.872368 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:12.873139 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:12.873570 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:12.873679 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:12.874338 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:12.874342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:12.874342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:46:12.874531 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:12.874570 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:46:12.874596 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:12.874656 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:12.874742 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 14:46:12.874825 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260620 14:46:12.875277 5757 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9b761b00c443e7a782eebc0bfc6b01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38649 } }
I20260620 14:46:12.875370 5757 leader_election.cc:302] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd9b761b00c443e7a782eebc0bfc6b01; no voters:
I20260620 14:46:12.875556 5757 leader_election.cc:288] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:46:12.875586 5762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:12.875715 5762 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01 [term 1 LEADER]: Becoming Leader. State: Replica: cd9b761b00c443e7a782eebc0bfc6b01, State: Running, Role: LEADER
I20260620 14:46:12.875797 5762 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9b761b00c443e7a782eebc0bfc6b01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38649 } }
I20260620 14:46:12.875804 5757 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:46:12.876178 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:46:12.876199 5763 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd9b761b00c443e7a782eebc0bfc6b01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9b761b00c443e7a782eebc0bfc6b01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38649 } } }
I20260620 14:46:12.876370 5763 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:12.876319 5764 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd9b761b00c443e7a782eebc0bfc6b01. Latest consensus state: current_term: 1 leader_uuid: "cd9b761b00c443e7a782eebc0bfc6b01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9b761b00c443e7a782eebc0bfc6b01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38649 } } }
I20260620 14:46:12.876579 5764 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:12.876665 5766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:46:12.877161 5766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:46:12.878520 5766 catalog_manager.cc:1347] Generated new cluster ID: e2c6c70b98354b24961d8581769c26dd
I20260620 14:46:12.878567 5766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:46:12.889065 5766 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:46:12.889552 5766 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:46:12.891187 5766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd9b761b00c443e7a782eebc0bfc6b01: Generated new TSK 0
I20260620 14:46:12.891338 5766 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:46:12.902460 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:12.899348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.0.98.62:38649)
0620 14:46:12.899400 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:12.899402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:12.899518 (+ 116us) server_negotiation.cc:407] Connection header received
0620 14:46:12.899583 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:12.899584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:12.899595 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:12.899656 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:46:12.899770 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:12.900256 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:12.901031 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:12.901230 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:12.901282 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:12.901446 (+ 164us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:46:12.902259 (+ 813us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44895/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:46:12.902385 (+ 126us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45678: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44895/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 14:46:12.902516 5760 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45678: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44895/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 14:46:12.902562 5786 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:12.899044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38649 (local address 127.0.0.1:45678)
0620 14:46:12.899455 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:12.899465 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:12.899525 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:12.899661 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:12.899661 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:12.899663 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:46:12.899715 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:12.899717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:12.900346 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:12.900347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:12.900958 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:12.900961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:12.901286 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:12.901294 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:12.901353 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:12.901358 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:46:12.902354 (+ 996us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44895/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:46:12.902456 (+ 102us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38649: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44895/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":75,"threads_started":1}
I20260620 14:46:12.902704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 5690
E20260620 14:46:12.913094 5787 webserver.cc:533] Webserver: sslize failed:
E20260620 14:46:12.913160 5787 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T14:46:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (191 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T14:46:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:46:12Z Disabled control of system clock
I20260620 14:46:12.945333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42727
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42727
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966748446763-392-0/testchainca.cert with env {}
W20260620 14:46:13.022001 5791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:13.022197 5791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:13.022225 5791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:13.023345 5791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:46:13.023391 5791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:13.023406 5791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:13.023442 5791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:46:13.023456 5791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:46:13.023473 5791 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:46:13.023494 5791 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 14:46:13.024933 5791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42727
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966748446763-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:13.025158 5791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:13.025343 5791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:13.029817 5799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:13.029832 5796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:13.029865 5791 server_base.cc:1034] running on GCE node
W20260620 14:46:13.029870 5797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:13.030225 5791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:13.030496 5791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:13.031625 5791 hybrid_clock.cc:648] HybridClock initialized: now 1781966773031612 us; error 28 us; skew 500 ppm
I20260620 14:46:13.032800 5791 webserver.cc:460] Webserver started at http://127.0.98.62:35907/ using document root <none> and password file <none>
I20260620 14:46:13.033010 5791 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:13.033053 5791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:13.033172 5791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:13.034155 5791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "2810bcc4d7b54a7aa0b2a1b300a99346"
format_stamp: "Formatted at 2026-06-20 14:46:13 on dist-test-slave-cx89"
server_key: "15d307479a78a7cb8c8338a988672732"
server_key_iv: "bd369a131cf6fb2470a4ac3025c174eb"
server_key_version: "encryptionkey@0"
I20260620 14:46:13.034459 5791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "2810bcc4d7b54a7aa0b2a1b300a99346"
format_stamp: "Formatted at 2026-06-20 14:46:13 on dist-test-slave-cx89"
server_key: "15d307479a78a7cb8c8338a988672732"
server_key_iv: "bd369a131cf6fb2470a4ac3025c174eb"
server_key_version: "encryptionkey@0"
I20260620 14:46:13.035655 5791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:46:13.036280 5805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:13.036435 5791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:13.036525 5791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "2810bcc4d7b54a7aa0b2a1b300a99346"
format_stamp: "Formatted at 2026-06-20 14:46:13 on dist-test-slave-cx89"
server_key: "15d307479a78a7cb8c8338a988672732"
server_key_iv: "bd369a131cf6fb2470a4ac3025c174eb"
server_key_version: "encryptionkey@0"
I20260620 14:46:13.036583 5791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:13.052412 5791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:13.052697 5791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:13.052804 5791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:13.056597 5791 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42727
I20260620 14:46:13.056627 5857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42727 every 8 connection(s)
I20260620 14:46:13.056957 5791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:46:13.057806 5858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:13.059392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 5791
I20260620 14:46:13.059509 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:46:13.059684 392 external_mini_cluster.cc:1428] Setting key 3ff92d6db0528de1a6a91283a24d0d18
I20260620 14:46:13.060103 5858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346: Bootstrap starting.
I20260620 14:46:13.060647 5858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:13.060906 5858 log.cc:824] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:13.061487 5858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346: No bootstrap required, opened a new log
I20260620 14:46:13.063134 5858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2810bcc4d7b54a7aa0b2a1b300a99346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42727 } }
I20260620 14:46:13.063259 5858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:13.063279 5858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2810bcc4d7b54a7aa0b2a1b300a99346, State: Initialized, Role: FOLLOWER
I20260620 14:46:13.063396 5858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2810bcc4d7b54a7aa0b2a1b300a99346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42727 } }
I20260620 14:46:13.063488 5858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:46:13.063512 5858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:46:13.063555 5858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:13.064064 5858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2810bcc4d7b54a7aa0b2a1b300a99346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42727 } }
I20260620 14:46:13.064167 5858 leader_election.cc:302] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2810bcc4d7b54a7aa0b2a1b300a99346; no voters:
I20260620 14:46:13.064316 5858 leader_election.cc:288] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:46:13.064349 5863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:13.064468 5863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346 [term 1 LEADER]: Becoming Leader. State: Replica: 2810bcc4d7b54a7aa0b2a1b300a99346, State: Running, Role: LEADER
I20260620 14:46:13.064563 5858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:46:13.064543 5863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2810bcc4d7b54a7aa0b2a1b300a99346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42727 } }
I20260620 14:46:13.064877 5863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2810bcc4d7b54a7aa0b2a1b300a99346. Latest consensus state: current_term: 1 leader_uuid: "2810bcc4d7b54a7aa0b2a1b300a99346" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2810bcc4d7b54a7aa0b2a1b300a99346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42727 } } }
I20260620 14:46:13.064946 5863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:13.064903 5865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2810bcc4d7b54a7aa0b2a1b300a99346" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2810bcc4d7b54a7aa0b2a1b300a99346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42727 } } }
I20260620 14:46:13.065084 5865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:13.065310 5861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:13.060158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48866 (local address 127.0.98.62:42727)
0620 14:46:13.060347 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:13.060355 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:13.060380 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:46:13.060856 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:13.060863 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:13.060926 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:13.061073 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:13.061281 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:13.062193 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:13.063082 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:13.063621 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:13.063734 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:13.064394 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:13.064399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:13.064399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:46:13.064599 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:13.064640 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:46:13.064655 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:13.064736 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:13.064907 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 14:46:13.065110 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
W20260620 14:46:13.066112 5879 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:46:13.066161 5879 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:46:13.066212 5869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:46:13.066354 5869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:46:13.066970 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:46:13.067637 5869 catalog_manager.cc:1347] Generated new cluster ID: d41d7c8e8fb2420cb63ef1b11c0f5dd1
I20260620 14:46:13.067687 5869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:46:13.076304 5869 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:46:13.076773 5869 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:46:13.080154 5869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2810bcc4d7b54a7aa0b2a1b300a99346: Generated new TSK 0
I20260620 14:46:13.080304 5869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:46:13.182688 5861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:13.087562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.0.98.62:42727)
0620 14:46:13.087609 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:13.087611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:13.087735 (+ 124us) server_negotiation.cc:407] Connection header received
0620 14:46:13.087804 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:13.087804 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:13.087813 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:13.087867 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:46:13.087998 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:13.088471 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:13.089572 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:13.089751 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:13.089794 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:13.089964 (+ 170us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:46:13.182480 (+ 92516us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:46:13.182629 (+ 149us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48874: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":26}
W20260620 14:46:13.182751 5861 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48874: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 14:46:13.182860 5887 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:13.087497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42727 (local address 127.0.0.1:48874)
0620 14:46:13.087666 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:13.087676 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:13.087740 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:13.087877 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:13.087878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:13.087880 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:46:13.087940 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:13.087942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:13.088534 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:13.088535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:13.089504 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:13.089507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:13.089811 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:13.089816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:13.089872 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:13.089878 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:46:13.182622 (+ 92744us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:46:13.182759 (+ 137us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42727: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260620 14:46:13.183255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 5791
2026-06-20T14:46:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (261 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (85 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:46:13 dist-test-slave-cx89 krb5kdc[5899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:46:13 dist-test-slave-cx89 krb5kdc[5899](info): set up 2 sockets
Jun 20 14:46:13 dist-test-slave-cx89 krb5kdc[5899](info): commencing operation
krb5kdc: starting...
W20260620 14:46:14.311434 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:46:14 dist-test-slave-cx89 krb5kdc[5899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:46:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:46:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:46:14.422590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45917
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:14.499811 5912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:14.500001 5912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:14.500029 5912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:14.501163 5912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:46:14.501202 5912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:14.501220 5912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:14.501237 5912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:46:14.501252 5912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:46:14.502749 5912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:14.502985 5912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:14.503185 5912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:14.507788 5917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:14.507822 5912 server_base.cc:1034] running on GCE node
W20260620 14:46:14.507799 5920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:14.507884 5918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:14.508191 5912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:14.508462 5912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:14.509601 5912 hybrid_clock.cc:648] HybridClock initialized: now 1781966774509585 us; error 29 us; skew 500 ppm
Jun 20 14:46:14 dist-test-slave-cx89 krb5kdc[5899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:14.511927 5912 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:46:14.512315 5912 webserver.cc:460] Webserver started at http://127.0.98.62:43387/ using document root <none> and password file <none>
I20260620 14:46:14.512519 5912 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:14.512563 5912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:14.512672 5912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:14.513680 5912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "a8bbda98a6834171b8287549983b91b1"
format_stamp: "Formatted at 2026-06-20 14:46:14 on dist-test-slave-cx89"
server_key: "690e667674fdec6c0ae3f7fe1afe531e"
server_key_iv: "f2d4c04cbe2bfdc136bac12fac0a008a"
server_key_version: "encryptionkey@0"
I20260620 14:46:14.513985 5912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "a8bbda98a6834171b8287549983b91b1"
format_stamp: "Formatted at 2026-06-20 14:46:14 on dist-test-slave-cx89"
server_key: "690e667674fdec6c0ae3f7fe1afe531e"
server_key_iv: "f2d4c04cbe2bfdc136bac12fac0a008a"
server_key_version: "encryptionkey@0"
I20260620 14:46:14.515167 5912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:46:14.515851 5927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:14.515990 5912 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:14.516067 5912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "a8bbda98a6834171b8287549983b91b1"
format_stamp: "Formatted at 2026-06-20 14:46:14 on dist-test-slave-cx89"
server_key: "690e667674fdec6c0ae3f7fe1afe531e"
server_key_iv: "f2d4c04cbe2bfdc136bac12fac0a008a"
server_key_version: "encryptionkey@0"
I20260620 14:46:14.516134 5912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:14.525570 5912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:14.527638 5912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:14.527768 5912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:14.531262 5912 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45917
I20260620 14:46:14.531298 5979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45917 every 8 connection(s)
I20260620 14:46:14.531764 5912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:46:14.532390 5980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:14.534502 5980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1: Bootstrap starting.
I20260620 14:46:14.535034 5980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:14.535261 5980 log.cc:824] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:14.535883 5980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1: No bootstrap required, opened a new log
I20260620 14:46:14.536861 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 5912
I20260620 14:46:14.536954 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:46:14.537142 392 external_mini_cluster.cc:1428] Setting key 43244c5c5ed7c64620c9ddd430d47934
I20260620 14:46:14.537693 5980 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8bbda98a6834171b8287549983b91b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45917 } }
I20260620 14:46:14.537859 5980 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:14.537897 5980 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8bbda98a6834171b8287549983b91b1, State: Initialized, Role: FOLLOWER
I20260620 14:46:14.538017 5980 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8bbda98a6834171b8287549983b91b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45917 } }
I20260620 14:46:14.538481 5980 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:46:14.538515 5980 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:46:14.538573 5980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:14.539189 5980 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8bbda98a6834171b8287549983b91b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45917 } }
I20260620 14:46:14.539311 5980 leader_election.cc:302] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8bbda98a6834171b8287549983b91b1; no voters:
I20260620 14:46:14.539498 5980 leader_election.cc:288] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:46:14.539570 5985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:14.539705 5985 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1 [term 1 LEADER]: Becoming Leader. State: Replica: a8bbda98a6834171b8287549983b91b1, State: Running, Role: LEADER
I20260620 14:46:14.539750 5980 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:46:14.539788 5985 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8bbda98a6834171b8287549983b91b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45917 } }
I20260620 14:46:14.540102 5985 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8bbda98a6834171b8287549983b91b1. Latest consensus state: current_term: 1 leader_uuid: "a8bbda98a6834171b8287549983b91b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8bbda98a6834171b8287549983b91b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45917 } } }
I20260620 14:46:14.540180 5985 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:14.540150 5986 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8bbda98a6834171b8287549983b91b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8bbda98a6834171b8287549983b91b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45917 } } }
I20260620 14:46:14.540307 5986 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:14.540941 5992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:46:14.541404 5992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:46:14.543473 5992 catalog_manager.cc:1347] Generated new cluster ID: 4427aeab86aa418eabe38a48b8bc32b6
I20260620 14:46:14.543524 5992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:46:14 dist-test-slave-cx89 krb5kdc[5899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:14.547063 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:14.537645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.0.98.62:45917)
0620 14:46:14.537798 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:14.537808 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:14.537870 (+ 62us) server_negotiation.cc:407] Connection header received
0620 14:46:14.538026 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:14.538042 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:14.538099 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:14.538262 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:14.538852 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:14.540235 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:14.541288 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:14.541779 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:14.541875 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:14.544330 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:14.544334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:14.544334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:14.544344 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:14.545773 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:14.546078 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:14.546080 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:14.546080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:14.546168 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:14.546333 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:14.546334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:14.546334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:14.546558 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:14.546722 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:14.546787 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 14:46:14.546913 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260620 14:46:14.557713 5992 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:46:14.558435 5992 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:46:14.561368 5992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8bbda98a6834171b8287549983b91b1: Generated new TSK 0
I20260620 14:46:14.561534 5992 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:46:14.590381 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45917
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:14.677846 6006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:14.678043 6006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:14.678066 6006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:14.679189 6006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:14.679229 6006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:14.679253 6006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:46:14.680708 6006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:14.680917 6006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:14.681131 6006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:14.685714 6012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:14.685798 6014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:14.685798 6011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:14.685994 6006 server_base.cc:1034] running on GCE node
I20260620 14:46:14.686275 6006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:14.686537 6006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:14.687703 6006 hybrid_clock.cc:648] HybridClock initialized: now 1781966774687687 us; error 29 us; skew 500 ppm
Jun 20 14:46:14 dist-test-slave-cx89 krb5kdc[5899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:14.689983 6006 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:46:14.690400 6006 webserver.cc:460] Webserver started at http://127.0.98.1:38627/ using document root <none> and password file <none>
I20260620 14:46:14.690594 6006 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:14.690634 6006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:14.690719 6006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:14.691820 6006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "e94a729530fd4e239daa50e2ea29489b"
format_stamp: "Formatted at 2026-06-20 14:46:14 on dist-test-slave-cx89"
server_key: "e028cfe2dd39fc55f6b7ae617d764747"
server_key_iv: "1b48a3365f6de36c9d795dfc73fcf87e"
server_key_version: "encryptionkey@0"
I20260620 14:46:14.692131 6006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e94a729530fd4e239daa50e2ea29489b"
format_stamp: "Formatted at 2026-06-20 14:46:14 on dist-test-slave-cx89"
server_key: "e028cfe2dd39fc55f6b7ae617d764747"
server_key_iv: "1b48a3365f6de36c9d795dfc73fcf87e"
server_key_version: "encryptionkey@0"
I20260620 14:46:14.693274 6006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:46:14.693912 6021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:14.694080 6006 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:14.694157 6006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "e94a729530fd4e239daa50e2ea29489b"
format_stamp: "Formatted at 2026-06-20 14:46:14 on dist-test-slave-cx89"
server_key: "e028cfe2dd39fc55f6b7ae617d764747"
server_key_iv: "1b48a3365f6de36c9d795dfc73fcf87e"
server_key_version: "encryptionkey@0"
I20260620 14:46:14.694208 6006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:14.705017 6006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:14.706694 6006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:14.706794 6006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:14.706976 6006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:14.707226 6006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:14.707254 6006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:14.707275 6006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:14.707291 6006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:14.712505 6006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46597
I20260620 14:46:14.712596 6134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46597 every 8 connection(s)
I20260620 14:46:14.712831 6006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:46:14.715060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 6006
I20260620 14:46:14.715167 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:46:14.715338 392 external_mini_cluster.cc:1428] Setting key ca02e5c8f713d67fdc9d844b575c6d6d
Jun 20 14:46:14 dist-test-slave-cx89 krb5kdc[5899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:14.722144 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:14.713465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33131 (local address 127.0.98.62:45917)
0620 14:46:14.713522 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:14.713524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:14.713713 (+ 189us) server_negotiation.cc:407] Connection header received
0620 14:46:14.714597 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:14.714598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:14.714607 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:14.714655 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:14.715374 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:14.716020 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:14.717364 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:14.717588 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:14.717645 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:14.719679 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:14.719680 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:14.719681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:14.719688 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:14.720934 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:14.721284 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:14.721286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:14.721286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:14.721345 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:14.721559 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:14.721560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:14.721560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:14.721671 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:14.721725 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:14.721974 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 14:46:14.722055 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:46:14.722335 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:14.713441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45917 (local address 127.0.98.1:33131)
0620 14:46:14.713599 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:14.713636 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:14.714497 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:14.714717 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:14.714720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:14.715004 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:14.715293 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:14.715300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:14.716103 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:14.716104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:14.717285 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:14.717288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:14.717656 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:14.717656 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:14.717782 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:14.718242 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:14.718250 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:14.719563 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:14.721006 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:14.721012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:14.721014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:14.721223 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:14.721396 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:14.721397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:14.721397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:14.721500 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:14.721722 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:14.721727 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:14.721861 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 14:46:14.721947 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:46:14.722126 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260620 14:46:14.722759 6135 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45917
I20260620 14:46:14.722865 6135 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:14.723026 6135 heartbeater.cc:511] Master 127.0.98.62:45917 requested a full tablet report, sending...
I20260620 14:46:14.723497 5944 ts_manager.cc:194] Registered new tserver with Master: e94a729530fd4e239daa50e2ea29489b (127.0.98.1:46597)
I20260620 14:46:14.724067 5944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:46:14.737469 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45917
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:14.816980 6140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:14.817170 6140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:14.817191 6140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:14.818262 6140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:14.818295 6140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:14.818318 6140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:46:14.819770 6140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:14.819950 6140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:14.820112 6140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:14.824692 6145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:14.824698 6146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:14.824704 6148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:14.824846 6140 server_base.cc:1034] running on GCE node
I20260620 14:46:14.825049 6140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:14.825255 6140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:14.826406 6140 hybrid_clock.cc:648] HybridClock initialized: now 1781966774826391 us; error 27 us; skew 500 ppm
Jun 20 14:46:14 dist-test-slave-cx89 krb5kdc[5899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:14.828752 6140 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:46:14.829159 6140 webserver.cc:460] Webserver started at http://127.0.98.2:45133/ using document root <none> and password file <none>
I20260620 14:46:14.829356 6140 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:14.829401 6140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:14.829517 6140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:14.830493 6140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "0f9f08e47699493aaa0e975ed8997101"
format_stamp: "Formatted at 2026-06-20 14:46:14 on dist-test-slave-cx89"
server_key: "807e0d063e622bf67bc662e785712a17"
server_key_iv: "51d46f6fd3903e20327d217148d1745e"
server_key_version: "encryptionkey@0"
I20260620 14:46:14.830806 6140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0f9f08e47699493aaa0e975ed8997101"
format_stamp: "Formatted at 2026-06-20 14:46:14 on dist-test-slave-cx89"
server_key: "807e0d063e622bf67bc662e785712a17"
server_key_iv: "51d46f6fd3903e20327d217148d1745e"
server_key_version: "encryptionkey@0"
I20260620 14:46:14.832072 6140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:46:14.832742 6155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:14.832896 6140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:46:14.832974 6140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "0f9f08e47699493aaa0e975ed8997101"
format_stamp: "Formatted at 2026-06-20 14:46:14 on dist-test-slave-cx89"
server_key: "807e0d063e622bf67bc662e785712a17"
server_key_iv: "51d46f6fd3903e20327d217148d1745e"
server_key_version: "encryptionkey@0"
I20260620 14:46:14.833031 6140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:14.842267 6140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:14.844321 6140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:14.844445 6140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:14.844628 6140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:14.844884 6140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:14.844915 6140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:14.844942 6140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:14.844964 6140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:14.850121 6140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36607
I20260620 14:46:14.850157 6268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36607 every 8 connection(s)
I20260620 14:46:14.850432 6140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:46:14.851524 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 6140
I20260620 14:46:14.851608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:46:14.851760 392 external_mini_cluster.cc:1428] Setting key aa54272c144801dc51ec48cdaf5b003d
Jun 20 14:46:14 dist-test-slave-cx89 krb5kdc[5899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:46:14.859753 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:14.851065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54367 (local address 127.0.98.62:45917)
0620 14:46:14.851107 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:14.851109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:14.851285 (+ 176us) server_negotiation.cc:407] Connection header received
0620 14:46:14.852301 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:14.852302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:14.852313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:14.852361 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:14.852958 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:14.853491 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:14.854474 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:14.854777 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:14.854842 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:14.857019 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:14.857021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:14.857021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:14.857030 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:14.858501 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:14.858877 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:14.858879 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:14.858879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:14.858933 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:14.859124 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:14.859125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:14.859125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:14.859252 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:14.859321 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:14.859607 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 14:46:14.859689 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:46:14.859874 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:14.851023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45917 (local address 127.0.98.2:54367)
0620 14:46:14.851172 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:14.851212 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:14.852214 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:14.852436 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:14.852440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:14.852693 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:14.852876 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:14.852881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:14.853592 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:14.853594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:14.854387 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:14.854389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:14.854851 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:14.854852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:14.854954 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:14.855484 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:14.855493 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:14.856933 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:14.858600 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:14.858605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:14.858607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:14.858807 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:14.858991 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:14.858992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:14.858992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:14.859074 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:14.859306 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:14.859309 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:14.859474 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 14:46:14.859567 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:46:14.859687 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260620 14:46:14.860298 6269 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45917
I20260620 14:46:14.860383 6269 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:14.860566 6269 heartbeater.cc:511] Master 127.0.98.62:45917 requested a full tablet report, sending...
I20260620 14:46:14.860889 5944 ts_manager.cc:194] Registered new tserver with Master: 0f9f08e47699493aaa0e975ed8997101 (127.0.98.2:36607)
I20260620 14:46:14.861227 5944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:46:14.876508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45917
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:14.953927 6274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:14.954130 6274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:14.954160 6274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:14.955284 6274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:14.955323 6274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:14.955358 6274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:46:14.956808 6274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:14.957026 6274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:14.957226 6274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:14.961795 6282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:14.961817 6280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:14.961812 6279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:14.961927 6274 server_base.cc:1034] running on GCE node
I20260620 14:46:14.962206 6274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:14.962386 6274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:14.963521 6274 hybrid_clock.cc:648] HybridClock initialized: now 1781966774963510 us; error 28 us; skew 500 ppm
Jun 20 14:46:14 dist-test-slave-cx89 krb5kdc[5899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:14.965714 6274 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:46:14.966135 6274 webserver.cc:460] Webserver started at http://127.0.98.3:41117/ using document root <none> and password file <none>
I20260620 14:46:14.966322 6274 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:14.966358 6274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:14.966452 6274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:14.967404 6274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "4b34bebf69c64beab847c6e4724512a9"
format_stamp: "Formatted at 2026-06-20 14:46:14 on dist-test-slave-cx89"
server_key: "ba455c8486077ece092671d0371c6ab8"
server_key_iv: "721944ffabc6ed61359150e43def5d39"
server_key_version: "encryptionkey@0"
I20260620 14:46:14.967741 6274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4b34bebf69c64beab847c6e4724512a9"
format_stamp: "Formatted at 2026-06-20 14:46:14 on dist-test-slave-cx89"
server_key: "ba455c8486077ece092671d0371c6ab8"
server_key_iv: "721944ffabc6ed61359150e43def5d39"
server_key_version: "encryptionkey@0"
I20260620 14:46:14.968879 6274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:46:14.969477 6289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:14.969609 6274 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:46:14.969679 6274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "4b34bebf69c64beab847c6e4724512a9"
format_stamp: "Formatted at 2026-06-20 14:46:14 on dist-test-slave-cx89"
server_key: "ba455c8486077ece092671d0371c6ab8"
server_key_iv: "721944ffabc6ed61359150e43def5d39"
server_key_version: "encryptionkey@0"
I20260620 14:46:14.969736 6274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:14.986410 6274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:14.988111 6274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:14.988236 6274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:14.988440 6274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:14.988714 6274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:14.988746 6274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:14.988775 6274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:14.988791 6274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:14.993898 6274 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39807
I20260620 14:46:14.993952 6402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39807 every 8 connection(s)
I20260620 14:46:14.994233 6274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:46:14 dist-test-slave-cx89 krb5kdc[5899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:15.000667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 6274
I20260620 14:46:15.000761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:46:15.000934 392 external_mini_cluster.cc:1428] Setting key 906f76aeac2d54e4230c5bfa1d364092
I20260620 14:46:15.002406 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:14.994882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52909 (local address 127.0.98.62:45917)
0620 14:46:14.994952 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:14.994954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:14.995084 (+ 130us) server_negotiation.cc:407] Connection header received
0620 14:46:14.995994 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:14.995995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:14.996005 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:14.996063 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:14.996612 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:14.997120 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:14.998041 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:14.998229 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:14.998277 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:15.000130 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:15.000132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:15.000132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:15.000139 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:15.001373 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:15.001739 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:15.001740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:15.001741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:15.001798 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:15.001969 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:15.001970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:15.001970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:15.002061 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:15.002105 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:15.002292 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 14:46:15.002345 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:46:15.002579 6405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:14.994848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45917 (local address 127.0.98.3:52909)
0620 14:46:14.994975 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:14.995013 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:14.995908 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:14.996119 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:14.996123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:14.996354 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:14.996544 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:14.996548 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:14.997193 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:14.997195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:14.997971 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:14.997973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:14.998283 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:14.998283 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:14.998373 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:14.998781 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:14.998788 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:15.000034 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:15.001428 (+ 1394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:15.001434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:15.001436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:15.001674 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:15.001839 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:15.001840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:15.001840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:15.001929 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:15.002097 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:15.002100 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:15.002203 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 14:46:15.002275 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:46:15.002383 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":64,"threads_started":1}
I20260620 14:46:15.002909 6403 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45917
I20260620 14:46:15.002985 6403 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:15.003144 6403 heartbeater.cc:511] Master 127.0.98.62:45917 requested a full tablet report, sending...
I20260620 14:46:15.003469 5944 ts_manager.cc:194] Registered new tserver with Master: 4b34bebf69c64beab847c6e4724512a9 (127.0.98.3:39807)
I20260620 14:46:15.003814 5944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52909
I20260620 14:46:15.012380 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/test-admin.keytab.
Jun 20 14:46:15 dist-test-slave-cx89 krb5kdc[5899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:15.027292 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 14:46:15.028762 6410 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:15.028119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37204 (local address 127.0.98.1:46597)
0620 14:46:15.028269 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:15.028273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:15.028283 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:46:15.028309 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:15.028311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:15.028323 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:15.028383 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:15.028686 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37204: BlockingRecv error: recv got EOF from 127.0.0.1:37204 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
W20260620 14:46:15.028837 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/krb5cc)
Jun 20 14:46:15 dist-test-slave-cx89 krb5kdc[5899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:15.029712 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:46:15 dist-test-slave-cx89 krb5kdc[5899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:15.030560 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:46:15 dist-test-slave-cx89 krb5kdc[5899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:46:15.037528 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:15.031105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37212 (local address 127.0.98.1:46597)
0620 14:46:15.031153 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:15.031155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:15.031166 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:46:15.031225 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:15.031226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:15.031235 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:15.031279 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:15.031668 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:15.032367 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:15.033066 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:15.033359 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:15.033405 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:15.035277 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:15.035284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:15.035284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:15.035290 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:15.036674 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:15.036962 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:15.036964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:15.036964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:15.037013 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:15.037147 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:15.037148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:15.037148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:15.037264 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:15.037367 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:15.037403 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 14:46:15.037458 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 14:46:15.039135 6410 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:15.038662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37220 (local address 127.0.98.1:46597)
0620 14:46:15.038719 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:15.038720 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:15.038746 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:46:15.038800 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:15.038800 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:15.038808 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:15.038838 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:15.039091 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37220: BlockingRecv error: recv got EOF from 127.0.0.1:37220 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 14:46:15.039208 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:46:15 dist-test-slave-cx89 krb5kdc[5899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:15.040158 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:46:15 dist-test-slave-cx89 krb5kdc[5899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:15.040942 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:46:15 dist-test-slave-cx89 krb5kdc[5899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:46:15.047000 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:15.041394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.0.98.1:46597)
0620 14:46:15.041444 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:15.041446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:15.041454 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:46:15.041509 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:15.041510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:15.041517 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:15.041572 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:15.041871 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:15.042344 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:15.042965 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:15.043170 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:15.043220 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:15.045037 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:15.045038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:15.045039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:15.045045 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:15.046082 (+ 1037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:15.046361 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:15.046362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:15.046362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:15.046410 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:15.046568 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:15.046569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:15.046569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:15.046655 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:15.046732 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:15.046827 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:46:15.046897 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 14:46:15.048331 6410 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:15.047873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37242 (local address 127.0.98.1:46597)
0620 14:46:15.047904 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:15.047905 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:15.047940 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:46:15.047981 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:15.047981 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:15.047988 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:15.048043 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:15.048291 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37242: BlockingRecv error: recv got EOF from 127.0.0.1:37242 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 14:46:15.048394 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966748446763-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:46:15 dist-test-slave-cx89 krb5kdc[5899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:15.049456 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:46:15 dist-test-slave-cx89 krb5kdc[5899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:15.050385 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:46:15 dist-test-slave-cx89 krb5kdc[5899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:46:15.056452 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:15.051029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37252 (local address 127.0.98.1:46597)
0620 14:46:15.051114 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:15.051116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:15.051124 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:46:15.051142 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:15.051142 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:15.051150 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:15.051206 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:15.051620 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:15.052072 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:15.052704 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:15.052883 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:15.052955 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:15.054688 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:15.054689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:15.054689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:15.054695 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:15.055728 (+ 1033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:15.055985 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:15.055985 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:15.055986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:15.056024 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:15.056149 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:15.056150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:15.056150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:15.056221 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:15.056279 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:15.056348 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:46:15.056394 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260620 14:46:15.056991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 6006
I20260620 14:46:15.061064 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 6140
I20260620 14:46:15.064819 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 6274
I20260620 14:46:15.068102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 5912
2026-06-20T14:46:15Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1710 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:46:15 dist-test-slave-cx89 krb5kdc[6423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:46:15 dist-test-slave-cx89 krb5kdc[6423](info): set up 2 sockets
Jun 20 14:46:15 dist-test-slave-cx89 krb5kdc[6423](info): commencing operation
krb5kdc: starting...
W20260620 14:46:16.015224 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:46:16 dist-test-slave-cx89 krb5kdc[6423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:46:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:46:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 14:46:16.123858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40459
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:16.200943 6436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:16.201150 6436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:16.201182 6436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:16.202297 6436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:46:16.202337 6436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:16.202354 6436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:16.202370 6436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:46:16.202386 6436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:46:16.203878 6436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:16.204116 6436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:16.204313 6436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:16.208765 6442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:16.208765 6444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:16.208870 6436 server_base.cc:1034] running on GCE node
W20260620 14:46:16.208796 6441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:16.209169 6436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:16.209452 6436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:16.210608 6436 hybrid_clock.cc:648] HybridClock initialized: now 1781966776210589 us; error 29 us; skew 500 ppm
Jun 20 14:46:16 dist-test-slave-cx89 krb5kdc[6423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:16.212849 6436 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 14:46:16.213228 6436 webserver.cc:460] Webserver started at http://127.0.98.62:34835/ using document root <none> and password file <none>
I20260620 14:46:16.213418 6436 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:16.213462 6436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:16.213589 6436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:16.214569 6436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "d08956433a1142789be562fbf8eb6245"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "92f1431b62f0f8a5ad12ad72d097835a"
server_key_iv: "3a61944663775c634fff831447b214e6"
server_key_version: "encryptionkey@0"
I20260620 14:46:16.214893 6436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "d08956433a1142789be562fbf8eb6245"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "92f1431b62f0f8a5ad12ad72d097835a"
server_key_iv: "3a61944663775c634fff831447b214e6"
server_key_version: "encryptionkey@0"
I20260620 14:46:16.216097 6436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:46:16.216703 6451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:16.216840 6436 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:16.216917 6436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "d08956433a1142789be562fbf8eb6245"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "92f1431b62f0f8a5ad12ad72d097835a"
server_key_iv: "3a61944663775c634fff831447b214e6"
server_key_version: "encryptionkey@0"
I20260620 14:46:16.216975 6436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:16.231926 6436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:16.233595 6436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:16.233724 6436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:16.237325 6436 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40459
I20260620 14:46:16.237366 6503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40459 every 8 connection(s)
I20260620 14:46:16.237689 6436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:46:16.237908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 6436
I20260620 14:46:16.238003 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:46:16.238180 392 external_mini_cluster.cc:1428] Setting key b8db693148dad28f87388758fabda970
I20260620 14:46:16.238792 6504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:16.241686 6504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245: Bootstrap starting.
I20260620 14:46:16.242261 6504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:16.242552 6504 log.cc:824] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:16.243188 6504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245: No bootstrap required, opened a new log
Jun 20 14:46:16 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:46:16.244985 6504 raft_consensus.cc:348] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } }
I20260620 14:46:16.245115 6504 raft_consensus.cc:374] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:16.245147 6504 raft_consensus.cc:729] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d08956433a1142789be562fbf8eb6245, State: Initialized, Role: FOLLOWER
I20260620 14:46:16.245267 6504 consensus_queue.cc:260] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } }
I20260620 14:46:16.245344 6504 raft_consensus.cc:388] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:46:16.245373 6504 raft_consensus.cc:482] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:46:16.245443 6504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:16.246037 6504 raft_consensus.cc:504] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } }
I20260620 14:46:16.246124 6504 leader_election.cc:302] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d08956433a1142789be562fbf8eb6245; no voters:
I20260620 14:46:16.246272 6504 leader_election.cc:288] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:46:16.246318 6509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:16.246485 6509 raft_consensus.cc:686] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [term 1 LEADER]: Becoming Leader. State: Replica: d08956433a1142789be562fbf8eb6245, State: Running, Role: LEADER
I20260620 14:46:16.246512 6504 sys_catalog.cc:564] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:46:16.246580 6509 consensus_queue.cc:237] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } }
I20260620 14:46:16.246904 6509 sys_catalog.cc:455] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d08956433a1142789be562fbf8eb6245. Latest consensus state: current_term: 1 leader_uuid: "d08956433a1142789be562fbf8eb6245" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } }
I20260620 14:46:16.246975 6509 sys_catalog.cc:458] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:16.247198 6516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:46:16.247624 6510 sys_catalog.cc:455] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d08956433a1142789be562fbf8eb6245" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } }
I20260620 14:46:16.247704 6516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:46:16.247696 6510 sys_catalog.cc:458] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:16.249033 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.238641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37674 (local address 127.0.98.62:40459)
0620 14:46:16.238826 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:16.238836 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:16.238868 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:46:16.239385 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:16.239394 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:16.239479 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:16.239635 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:16.240177 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.241460 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.242212 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.242812 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.242913 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.245022 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:16.245026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:16.245026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:16.245038 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:16.246807 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.247221 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.247225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.247225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.247326 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.247586 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.247587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.247588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.247890 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:16.248028 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:16.248711 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 14:46:16.248844 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260620 14:46:16.249282 6516 catalog_manager.cc:1347] Generated new cluster ID: 61a4b4a0e1f64df3b241f2a80d00c0c9
I20260620 14:46:16.249327 6516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:46:16.261031 6516 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:46:16.261518 6516 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:46:16.270562 6516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245: Generated new TSK 0
I20260620 14:46:16.270723 6516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 14:46:16.302045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40459
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:16.384544 6530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:16.384747 6530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:16.384768 6530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:16.385861 6530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:16.385896 6530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:16.385921 6530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:46:16.387282 6530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:16.387501 6530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:16.387682 6530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:16.392323 6538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:16.392377 6536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:16.392445 6530 server_base.cc:1034] running on GCE node
W20260620 14:46:16.392335 6535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:16.392678 6530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:16.392916 6530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:16.394054 6530 hybrid_clock.cc:648] HybridClock initialized: now 1781966776394043 us; error 30 us; skew 500 ppm
Jun 20 14:46:16 dist-test-slave-cx89 krb5kdc[6423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:16.403892 6530 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:46:16.404369 6530 webserver.cc:460] Webserver started at http://127.0.98.1:44473/ using document root <none> and password file <none>
I20260620 14:46:16.404578 6530 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:16.404626 6530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:16.404749 6530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:16.405766 6530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "7f50bab07914483c9672fb1b82ea9a3f"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "fce99534ae70f5e9599d4557b8d35bbe"
server_key_iv: "485a724a0976f7a5371f3754f51fcf2b"
server_key_version: "encryptionkey@0"
I20260620 14:46:16.406095 6530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7f50bab07914483c9672fb1b82ea9a3f"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "fce99534ae70f5e9599d4557b8d35bbe"
server_key_iv: "485a724a0976f7a5371f3754f51fcf2b"
server_key_version: "encryptionkey@0"
I20260620 14:46:16.407280 6530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:46:16.407967 6545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:16.408114 6530 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:16.408195 6530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "7f50bab07914483c9672fb1b82ea9a3f"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "fce99534ae70f5e9599d4557b8d35bbe"
server_key_iv: "485a724a0976f7a5371f3754f51fcf2b"
server_key_version: "encryptionkey@0"
I20260620 14:46:16.408259 6530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:16.426262 6530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:16.428352 6530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:16.428479 6530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:16.428681 6530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:16.428946 6530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:16.428977 6530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:16.429014 6530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:16.429039 6530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:16.434249 6530 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38171
I20260620 14:46:16.434304 6658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38171 every 8 connection(s)
I20260620 14:46:16.434580 6530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:46:16.436921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 6530
I20260620 14:46:16.437014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:46:16.437191 392 external_mini_cluster.cc:1428] Setting key d6c3bf1e845adfc373b76f7d92f97194
Jun 20 14:46:16 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:46:16.443296 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.435193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48377 (local address 127.0.98.62:40459)
0620 14:46:16.435261 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:16.435263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:16.435457 (+ 194us) server_negotiation.cc:407] Connection header received
0620 14:46:16.435935 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:16.435936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:16.435947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:16.435994 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:16.436877 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.437539 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.438558 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.438788 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.438850 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.440920 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:16.440921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:16.440922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:16.440928 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:16.442124 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.442466 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.442467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.442468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.442515 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.442725 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.442726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.442726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.442826 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:16.442879 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:16.443124 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 14:46:16.443208 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:46:16.443475 6661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.435179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:48377)
0620 14:46:16.435331 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:16.435366 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:16.435861 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:16.436056 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:16.436059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:16.436577 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:16.436765 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:16.436770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.437641 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.437641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:16.438471 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:16.438474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.438890 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.438891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:16.439020 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.439538 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:16.439546 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:16.440834 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:16.442195 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:16.442201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:16.442203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:16.442405 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:16.442570 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:16.442571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:16.442571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:16.442674 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:16.442877 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:16.442883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:16.443014 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 14:46:16.443100 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:46:16.443249 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260620 14:46:16.443925 6659 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40459
I20260620 14:46:16.444008 6659 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:16.444180 6659 heartbeater.cc:511] Master 127.0.98.62:40459 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:46:16.444649 6468 ts_manager.cc:194] Registered new tserver with Master: 7f50bab07914483c9672fb1b82ea9a3f (127.0.98.1:38171)
I20260620 14:46:16.445233 6468 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 14:46:16.461498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40459
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:16.540851 6664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:16.541049 6664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:16.541071 6664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:16.542160 6664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:16.542197 6664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:16.542222 6664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:46:16.543617 6664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:16.543795 6664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:16.543974 6664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:16.548743 6672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:16.548840 6669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:16.548923 6670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:16.549310 6664 server_base.cc:1034] running on GCE node
I20260620 14:46:16.549499 6664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:16.549718 6664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:16.550864 6664 hybrid_clock.cc:648] HybridClock initialized: now 1781966776550856 us; error 33 us; skew 500 ppm
Jun 20 14:46:16 dist-test-slave-cx89 krb5kdc[6423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:16.553227 6664 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:46:16.553654 6664 webserver.cc:460] Webserver started at http://127.0.98.2:36821/ using document root <none> and password file <none>
I20260620 14:46:16.553841 6664 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:16.553886 6664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:16.554005 6664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:16.555001 6664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "e07ea2005fa84020a75becdaa5468e7d"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "de37e2d42de54c438b732f8804eb44ce"
server_key_iv: "7305367bf795968bee38ea50ef971a8e"
server_key_version: "encryptionkey@0"
I20260620 14:46:16.555320 6664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e07ea2005fa84020a75becdaa5468e7d"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "de37e2d42de54c438b732f8804eb44ce"
server_key_iv: "7305367bf795968bee38ea50ef971a8e"
server_key_version: "encryptionkey@0"
I20260620 14:46:16.556522 6664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:46:16.557238 6679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:16.557641 6664 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:46:16.557729 6664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "e07ea2005fa84020a75becdaa5468e7d"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "de37e2d42de54c438b732f8804eb44ce"
server_key_iv: "7305367bf795968bee38ea50ef971a8e"
server_key_version: "encryptionkey@0"
I20260620 14:46:16.557791 6664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:16.568851 6664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:16.570891 6664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:16.571007 6664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:16.571213 6664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:16.571542 6664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:16.571576 6664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:16.571599 6664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:16.571630 6664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:16.576874 6664 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40311
I20260620 14:46:16.576911 6792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40311 every 8 connection(s)
I20260620 14:46:16.577199 6664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:46:16 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:46:16.585134 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.577877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51401 (local address 127.0.98.62:40459)
0620 14:46:16.577916 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:16.577918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:16.578096 (+ 178us) server_negotiation.cc:407] Connection header received
0620 14:46:16.578539 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:16.578540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:16.578551 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:16.578600 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:16.579518 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.580009 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.580934 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.581130 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.581189 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.583012 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:16.583013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:16.583013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:16.583020 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:16.584108 (+ 1088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.584417 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.584418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.584418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.584459 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.584620 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.584620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.584621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.584708 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:16.584752 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:16.585011 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 14:46:16.585072 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:46:16.585348 6795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.577847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:51401)
0620 14:46:16.577990 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:16.578023 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:16.578463 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:16.578663 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:16.578666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:16.579182 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:16.579372 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:16.579377 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.580071 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.580072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:16.580853 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:16.580856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.581194 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.581195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:16.581285 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.581729 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:16.581737 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:16.582936 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:16.584167 (+ 1231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:16.584172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:16.584173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:16.584360 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:16.584498 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:16.584499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:16.584499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:16.584580 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:16.584777 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:16.584782 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:16.584904 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 14:46:16.584996 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 14:46:16.585116 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260620 14:46:16.585819 6793 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40459
I20260620 14:46:16.585909 6793 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:16.585958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 6664
I20260620 14:46:16.586045 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:46:16.586102 6793 heartbeater.cc:511] Master 127.0.98.62:40459 requested a full tablet report, sending...
I20260620 14:46:16.586218 392 external_mini_cluster.cc:1428] Setting key f41dc8fe07cf6669a15905a22ec16ee4
I20260620 14:46:16.586452 6468 ts_manager.cc:194] Registered new tserver with Master: e07ea2005fa84020a75becdaa5468e7d (127.0.98.2:40311)
I20260620 14:46:16.586951 6468 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51401
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 14:46:16.609869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40459
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:16.689170 6798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:16.689375 6798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:16.689397 6798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:16.690477 6798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:16.690510 6798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:16.690534 6798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:46:16.692202 6798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:16.692400 6798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:16.692584 6798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:16.697631 6803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:16.697686 6804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:16.697655 6798 server_base.cc:1034] running on GCE node
W20260620 14:46:16.697813 6806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:16.698081 6798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:16.698319 6798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:16.699461 6798 hybrid_clock.cc:648] HybridClock initialized: now 1781966776699446 us; error 34 us; skew 500 ppm
Jun 20 14:46:16 dist-test-slave-cx89 krb5kdc[6423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:16.701882 6798 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:46:16.702310 6798 webserver.cc:460] Webserver started at http://127.0.98.3:43397/ using document root <none> and password file <none>
I20260620 14:46:16.702514 6798 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:16.702565 6798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:16.702670 6798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:16.703779 6798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "a7332d57250a4c4da5689a2dec888ff3"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "655212978fd79628017e58672dd47786"
server_key_iv: "fc0e6560a7f640835304b6ed98fbafa8"
server_key_version: "encryptionkey@0"
I20260620 14:46:16.704099 6798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a7332d57250a4c4da5689a2dec888ff3"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "655212978fd79628017e58672dd47786"
server_key_iv: "fc0e6560a7f640835304b6ed98fbafa8"
server_key_version: "encryptionkey@0"
I20260620 14:46:16.705281 6798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:46:16.705907 6813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:16.706034 6798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:16.706120 6798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "a7332d57250a4c4da5689a2dec888ff3"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "655212978fd79628017e58672dd47786"
server_key_iv: "fc0e6560a7f640835304b6ed98fbafa8"
server_key_version: "encryptionkey@0"
I20260620 14:46:16.706179 6798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:16.715271 6798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:16.716923 6798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:16.717046 6798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:16.717245 6798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:16.717510 6798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:16.717541 6798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:16.717577 6798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:16.717602 6798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:16.722945 6798 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40747
I20260620 14:46:16.722992 6926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40747 every 8 connection(s)
I20260620 14:46:16.723273 6798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:46:16.724174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 6798
I20260620 14:46:16.724268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:46:16.724434 392 external_mini_cluster.cc:1428] Setting key 4f7838bda5fdbc022b54724d07fe5dac
Jun 20 14:46:16 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:46:16.732273 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.723949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38041 (local address 127.0.98.62:40459)
0620 14:46:16.724027 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:16.724029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:16.724205 (+ 176us) server_negotiation.cc:407] Connection header received
0620 14:46:16.724768 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:16.724770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:16.724782 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:16.724839 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:16.725786 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.726351 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.727308 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.727555 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.727626 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.729835 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:16.729836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:16.729837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:16.729843 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:16.731055 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.731401 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.731402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.731402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.731494 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.731764 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.731764 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.731765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.731875 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:16.731928 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:16.732151 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 14:46:16.732209 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:46:16.732468 6929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.723929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:38041)
0620 14:46:16.724077 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:16.724123 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:16.724667 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:16.724902 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:16.724906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:16.725512 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:16.725708 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:16.725713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.726431 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.726433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:16.727223 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:16.727225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.727707 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.727708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:16.727825 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.728334 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:16.728342 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:16.729751 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:16.731122 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:16.731127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:16.731138 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:16.731347 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:16.731559 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:16.731560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:16.731560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:16.731701 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:16.731923 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:16.731926 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:16.732045 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 14:46:16.732134 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:46:16.732257 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260620 14:46:16.732915 6927 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40459
I20260620 14:46:16.732995 6927 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:16.733173 6927 heartbeater.cc:511] Master 127.0.98.62:40459 requested a full tablet report, sending...
I20260620 14:46:16.733526 6468 ts_manager.cc:194] Registered new tserver with Master: a7332d57250a4c4da5689a2dec888ff3 (127.0.98.3:40747)
I20260620 14:46:16.733880 6468 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38041
I20260620 14:46:16.735998 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260620 14:46:16.758399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35981
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40459,127.0.98.61:35981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:16.837347 6932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:16.837545 6932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:16.837566 6932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:16.838653 6932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:46:16.838693 6932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:16.838706 6932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:16.838716 6932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:46:16.838727 6932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:46:16.840154 6932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40459,127.0.98.61:35981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
W20260620 14:46:16.840250 6932 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:40459,127.0.98.61:35981'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 14:46:16.840389 6932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:16.840592 6932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:16.845141 6937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:16.845168 6938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:16.845208 6932 server_base.cc:1034] running on GCE node
W20260620 14:46:16.845168 6940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:16.845541 6932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:16.845801 6932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:16.846928 6932 hybrid_clock.cc:648] HybridClock initialized: now 1781966776846917 us; error 31 us; skew 500 ppm
Jun 20 14:46:16 dist-test-slave-cx89 krb5kdc[6423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:16.849282 6932 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 14:46:16.849645 6932 webserver.cc:460] Webserver started at http://127.0.98.61:39083/ using document root <none> and password file <none>
I20260620 14:46:16.849809 6932 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:16.849846 6932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:16.849934 6932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:16.850898 6932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/data/instance:
uuid: "6604b93b1d5b4df4963aea5f8e50a62a"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "a86ca39ab79ce01c16490bb14d7a9b49"
server_key_iv: "24d128e4142be73aacc7bbab7a3e81f1"
server_key_version: "encryptionkey@0"
I20260620 14:46:16.851212 6932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/wal/instance:
uuid: "6604b93b1d5b4df4963aea5f8e50a62a"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "a86ca39ab79ce01c16490bb14d7a9b49"
server_key_iv: "24d128e4142be73aacc7bbab7a3e81f1"
server_key_version: "encryptionkey@0"
I20260620 14:46:16.852449 6932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:46:16.853120 6947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:16.853272 6932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:16.853350 6932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/wal
uuid: "6604b93b1d5b4df4963aea5f8e50a62a"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "a86ca39ab79ce01c16490bb14d7a9b49"
server_key_iv: "24d128e4142be73aacc7bbab7a3e81f1"
server_key_version: "encryptionkey@0"
I20260620 14:46:16.853399 6932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:16.861688 6932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:16.863670 6932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:16.863770 6932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:16.867141 6932 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35981
I20260620 14:46:16.867174 6999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35981 every 8 connection(s)
I20260620 14:46:16.867484 6932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:46:16.868340 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:16.869709 7000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:46:16.873190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 6932
Jun 20 14:46:16 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 14:46:16 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:46:16.877934 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.870335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37678 (local address 127.0.98.62:40459)
0620 14:46:16.870378 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:16.870380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:16.870553 (+ 173us) server_negotiation.cc:407] Connection header received
0620 14:46:16.871034 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:16.871035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:16.871047 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:16.871097 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:16.871720 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.872216 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.873173 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.873388 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.873442 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.875378 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:16.875380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:16.875380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:16.875387 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:16.876690 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.877029 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.877031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.877031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.877086 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.877288 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.877289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.877289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.877463 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:16.877532 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:16.877773 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 14:46:16.877846 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:46:16.878013 7002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.870285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.0.1:37678)
0620 14:46:16.870432 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:16.870471 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:16.870926 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:16.871171 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:16.871175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:16.871399 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:16.871642 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:16.871647 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.872292 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.872294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:16.873096 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:16.873098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.873452 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.873453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:16.873575 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.874035 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:16.874043 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:16.875301 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:16.876757 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:16.876763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:16.876765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:16.876966 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:16.877137 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:16.877138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:16.877138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:16.877236 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:16.877531 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:16.877537 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:16.877655 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 14:46:16.877738 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:46:16.877846 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
I20260620 14:46:16.878578 7000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:46:16.881090 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.873622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44844 (local address 127.0.98.61:35981)
0620 14:46:16.873760 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:16.873764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:16.873789 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:46:16.873830 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:16.873832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:16.873851 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:16.873913 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:16.874462 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.875176 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.875911 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.876207 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.876264 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.878433 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:16.878439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:16.878439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:16.878446 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:16.880107 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.880441 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.880443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.880443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.880488 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.880641 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.880642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.880642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.880748 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:16.880847 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:16.880929 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:46:16.880996 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
Jun 20 14:46:16 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:46:16.885118 7002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.878808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35981 (local address 127.0.0.1:44858)
0620 14:46:16.878858 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:16.878867 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:16.878899 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:16.879091 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:16.879092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:16.879204 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:16.879256 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:16.879258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.880114 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.880115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:16.880786 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:16.880788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.881217 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.881218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:16.881281 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.881578 (+ 297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:16.881582 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:16.882578 (+ 996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:16.884275 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:16.884276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:16.884276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:16.884413 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:16.884567 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:16.884567 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:16.884568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:16.884615 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:16.884820 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:16.884820 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:16.884854 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 14:46:16.884908 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:46:16.884994 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 14:46:16.885185 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.878862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44858 (local address 127.0.98.61:35981)
0620 14:46:16.878987 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:16.878989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:16.878994 (+ 5us) server_negotiation.cc:407] Connection header received
0620 14:46:16.879009 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:16.879010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:16.879018 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:16.879088 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:16.879322 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.880034 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.880853 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.881153 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.881208 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.882642 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:16.882643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:16.882644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:16.882649 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:16.884211 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.884467 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.884467 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.884468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.884513 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.884662 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.884663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.884663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.884771 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:16.884821 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:16.884942 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 14:46:16.885042 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260620 14:46:16.886332 7000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: Bootstrap starting.
I20260620 14:46:16.886893 7000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:16.887133 7000 log.cc:824] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:16.887799 7000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: No bootstrap required, opened a new log
I20260620 14:46:16.889083 7000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } }
I20260620 14:46:16.889195 7000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:16.889215 7000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6604b93b1d5b4df4963aea5f8e50a62a, State: Initialized, Role: FOLLOWER
I20260620 14:46:16.889312 7000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } }
I20260620 14:46:16.889565 7000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:46:16.889551 7007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } } }
I20260620 14:46:16.889655 7007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:46:16.890969 7018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:46:16.891022 7018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:46:16.892346 6932 master_runner.cc:428] Detected that this master 6604b93b1d5b4df4963aea5f8e50a62a is joining an existing cluster
I20260620 14:46:16.892382 6932 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:35981
I20260620 14:46:16.898953 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.893358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44870 (local address 127.0.98.61:35981)
0620 14:46:16.893390 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:16.893392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:16.893400 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:46:16.893453 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:16.893454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:16.893463 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:16.893522 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:16.893830 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.894358 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.895045 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.895250 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.895300 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.896239 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:16.896240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:16.896240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:16.896246 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:16.897775 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.898119 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.898120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.898120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.898175 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.898488 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.898489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.898489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.898605 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:16.898659 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:16.898813 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 14:46:16.898884 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:46:16.899250 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.893192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35981 (local address 127.0.0.1:44870)
0620 14:46:16.893344 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:16.893355 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:16.893406 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:16.893530 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:16.893531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:16.893707 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:16.893773 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:16.893774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.894437 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.894437 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:16.894977 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:16.894978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.895315 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.895315 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:16.895371 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.895780 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:16.895785 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:16.896156 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:16.897842 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:16.897844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:16.897844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:16.898049 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:16.898238 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:16.898238 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:16.898239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:16.898298 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:16.898658 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:16.898659 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:16.898706 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:46:16.899067 (+ 361us) client_negotiation.cc:240] Negotiation successful
0620 14:46:16.899155 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260620 14:46:16.899479 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.893381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37694 (local address 127.0.98.62:40459)
0620 14:46:16.893422 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:16.893423 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:16.893543 (+ 120us) server_negotiation.cc:407] Connection header received
0620 14:46:16.893615 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:16.893616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:16.893626 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:16.893676 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:16.894016 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.894766 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.895830 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.896109 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.896180 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.897036 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:16.897038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:16.897038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:16.897046 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:16.898353 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.898707 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.898708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.898709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.898762 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.898950 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.898951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.898951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.899072 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:16.899130 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:16.899303 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 14:46:16.899374 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:46:16.899847 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.893198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.0.1:37694)
0620 14:46:16.893495 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:16.893503 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:16.893565 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:16.893689 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:16.893689 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:16.893857 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:16.893929 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:16.893930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.894857 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.894858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:16.895724 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:16.895726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.896184 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.896185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:16.896246 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.896593 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:16.896597 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:16.896960 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:16.898418 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:16.898418 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:16.898419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:16.898612 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:16.898820 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:16.898821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:16.898821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:16.898888 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:16.899158 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:16.899158 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:16.899196 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:46:16.899590 (+ 394us) client_negotiation.cc:240] Negotiation successful
0620 14:46:16.899751 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"spinlock_wait_cycles":384,"thread_start_us":88,"threads_started":1}
Jun 20 14:46:16 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:46:16.908640 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.901724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44876 (local address 127.0.98.61:35981)
0620 14:46:16.901759 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:16.901760 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:16.901948 (+ 188us) server_negotiation.cc:407] Connection header received
0620 14:46:16.902028 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:16.902029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:16.902042 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:16.902090 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:16.902546 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.903012 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.904132 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.904310 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.904363 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.906281 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:16.906283 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:16.906283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:16.906290 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:16.907596 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.907932 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.907933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.907933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.907982 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.908146 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.908147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.908147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.908244 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:16.908294 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:16.908453 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 14:46:16.908524 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:46:16.908680 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.901691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35981 (local address 127.0.0.1:44876)
0620 14:46:16.901840 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:16.901885 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:16.901960 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:16.902096 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:16.902099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:16.902416 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:16.902471 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:16.902473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.903091 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.903092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:16.904036 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:16.904038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.904372 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.904373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:16.904443 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.904828 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:16.904833 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:16.906209 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:16.907674 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:16.907681 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:16.907683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:16.907871 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:16.908032 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:16.908032 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:16.908032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:16.908089 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:16.908294 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:16.908302 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:16.908353 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 14:46:16.908425 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:46:16.908589 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260620 14:46:16.909198 6468 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:35981: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 35981
}
attrs {
promote: true
}
}
dest_uuid: "d08956433a1142789be562fbf8eb6245"
cas_config_opid_index: -1
I20260620 14:46:16.909475 6468 consensus_queue.cc:237] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: true } }
I20260620 14:46:16.909890 7028 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6604b93b1d5b4df4963aea5f8e50a62a (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: true } } }
I20260620 14:46:16.910221 6974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:16.910727 7028 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:35981
I20260620 14:46:16.910799 7028 sys_catalog.cc:455] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d08956433a1142789be562fbf8eb6245" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: true } } }
I20260620 14:46:16.910852 7028 sys_catalog.cc:458] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:16.910929 6932 master.cc:537] Master@127.0.98.61:35981 shutting down...
I20260620 14:46:16.911082 6974 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Refusing update from remote peer d08956433a1142789be562fbf8eb6245: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 14:46:16.911373 7027 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:46:16.912068 6454 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 14:46:16.912842 6454 consensus_peers.cc:487] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 -> Peer 6604b93b1d5b4df4963aea5f8e50a62a (127.0.98.61:35981): Couldn't send request to peer 6604b93b1d5b4df4963aea5f8e50a62a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:46:16.913805 6932 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:46:16.913877 6932 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:16.913923 6932 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: stopping tablet replica
I20260620 14:46:16.926714 6932 master.cc:552] Master@127.0.98.61:35981 shutdown complete.
I20260620 14:46:16.926769 6932 master_runner.cc:305] Clearing existing system tablet
I20260620 14:46:16.926891 6932 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:16.926936 6932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:16.927696 6932 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:46:16.928138 7032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:16.928256 6932 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:16.928311 6932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:16.928334 6932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/wal
uuid: "6604b93b1d5b4df4963aea5f8e50a62a"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "a86ca39ab79ce01c16490bb14d7a9b49"
server_key_iv: "24d128e4142be73aacc7bbab7a3e81f1"
server_key_version: "encryptionkey@0"
I20260620 14:46:16.929071 6932 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:16.930631 6932 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:46:16.930696 6932 log.cc:1196] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 14:46:16.930867 6932 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: Deleting consensus metadata
I20260620 14:46:16.930971 6932 master_runner.cc:315] Copying system tablet from 127.0.98.62:40459
I20260620 14:46:16.931560 6932 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40459
I20260620 14:46:16.937358 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.931884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37704 (local address 127.0.98.62:40459)
0620 14:46:16.931927 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:16.931929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:16.932036 (+ 107us) server_negotiation.cc:407] Connection header received
0620 14:46:16.932128 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:16.932129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:16.932139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:16.932184 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:16.932489 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.932971 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.934002 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.934173 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.934214 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.935164 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:16.935165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:16.935166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:16.935172 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:16.936441 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.936777 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.936778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.936778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.936820 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.936979 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.936979 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.936980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.937065 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:16.937121 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:16.937248 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 14:46:16.937300 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:46:16.937429 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.931830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.0.1:37704)
0620 14:46:16.931973 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:16.931982 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:16.932068 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:16.932189 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:16.932190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:16.932369 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:16.932432 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:16.932434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.933033 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.933034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:16.933928 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:16.933931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.934228 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.934229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:16.934294 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.934687 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:16.934692 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:16.935074 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:16.936507 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:16.936508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:16.936509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:16.936714 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:16.936871 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:16.936871 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:16.936872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:16.936930 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:16.937119 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:16.937119 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:16.937163 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 14:46:16.937232 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:46:16.937324 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260620 14:46:16.937755 6488 tablet_copy_service.cc:140] P d08956433a1142789be562fbf8eb6245: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6604b93b1d5b4df4963aea5f8e50a62a ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:37704)
I20260620 14:46:16.937836 6488 tablet_copy_service.cc:161] P d08956433a1142789be562fbf8eb6245: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6604b93b1d5b4df4963aea5f8e50a62a at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:37704: session id = 6604b93b1d5b4df4963aea5f8e50a62a-00000000000000000000000000000000
I20260620 14:46:16.938413 6488 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:46:16.938953 6932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:46:16.940142 6932 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: tablet copy: Starting download of 0 data blocks...
I20260620 14:46:16.940279 6932 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: tablet copy: Starting download of 1 WAL segments...
I20260620 14:46:16.941011 6932 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:46:16.941815 6488 tablet_copy_service.cc:342] P d08956433a1142789be562fbf8eb6245: Request end of tablet copy session 6604b93b1d5b4df4963aea5f8e50a62a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:37704
I20260620 14:46:16.941870 6488 tablet_copy_service.cc:433] P d08956433a1142789be562fbf8eb6245: ending tablet copy session 6604b93b1d5b4df4963aea5f8e50a62a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6604b93b1d5b4df4963aea5f8e50a62a
I20260620 14:46:16.943867 6932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:16.944835 7040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:16.944962 7039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:16.948146 7042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:16.948287 6932 server_base.cc:1034] running on GCE node
I20260620 14:46:16.948390 6932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:16.948509 6932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:16.949620 6932 hybrid_clock.cc:648] HybridClock initialized: now 1781966776949611 us; error 25 us; skew 500 ppm
Jun 20 14:46:16 dist-test-slave-cx89 krb5kdc[6423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:16.950862 6932 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 14:46:16.951129 6932 webserver.cc:460] Webserver started at http://127.0.98.61:40941/ using document root <none> and password file <none>
I20260620 14:46:16.951232 6932 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:16.951263 6932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:16.951922 6932 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:16.952411 7049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:16.952520 6932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:16.952567 6932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/wal
uuid: "6604b93b1d5b4df4963aea5f8e50a62a"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "a86ca39ab79ce01c16490bb14d7a9b49"
server_key_iv: "24d128e4142be73aacc7bbab7a3e81f1"
server_key_version: "encryptionkey@0"
I20260620 14:46:16.952620 6932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:16.974520 6932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:16.976213 6932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:16.979166 6932 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35981
I20260620 14:46:16.979194 7101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35981 every 8 connection(s)
I20260620 14:46:16.979519 6932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:46:16.980211 7102 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:46:16.980584 7102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: Bootstrap starting.
I20260620 14:46:16.982733 7102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:46:16.982924 7102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: Bootstrap complete.
I20260620 14:46:16.983098 7102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: true } }
I20260620 14:46:16.983165 7102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6604b93b1d5b4df4963aea5f8e50a62a, State: Initialized, Role: LEARNER
I20260620 14:46:16.983201 7102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: true } }
I20260620 14:46:16.983356 7102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:46:16.984735 7117 catalog_manager.cc:1259] Loaded cluster ID: 61a4b4a0e1f64df3b241f2a80d00c0c9
I20260620 14:46:16.984812 7117 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: loading cluster ID for follower catalog manager: success
I20260620 14:46:16.984442 7106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: true } } }
I20260620 14:46:16.985417 7106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [sys.catalog]: This master's current role is: LEARNER
I20260620 14:46:16.986001 7117 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: acquiring CA information for follower catalog manager: success
I20260620 14:46:16.986325 7117 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:46:16.989670 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:16.982419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.0.98.61:35981)
0620 14:46:16.982551 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:16.982553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:16.982562 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:46:16.982585 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:16.982586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:16.982597 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:16.982683 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:16.983326 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.984372 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.985468 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:16.985755 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:16.985809 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:16.987186 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:16.987187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:16.987187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:16.987193 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:16.988602 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.988967 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.988968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.988968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.989012 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:16.989194 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:16.989194 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:16.989195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:16.989297 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:16.989379 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:16.989503 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 14:46:16.989587 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260620 14:46:17.014043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35679
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40459,127.0.98.61:35981,127.0.98.60:35679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:17.099326 7120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:17.099539 7120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:17.099560 7120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:17.100667 7120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:46:17.100706 7120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:17.100719 7120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:17.100742 7120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:46:17.100765 7120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:46:17.102183 7120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40459,127.0.98.61:35981,127.0.98.60:35679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:17.102394 7120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:17.102588 7120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:17.106946 7126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:17.107038 7125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:17.106937 7128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:17.107041 7120 server_base.cc:1034] running on GCE node
I20260620 14:46:17.107460 7120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:17.107743 7120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:17.108881 7120 hybrid_clock.cc:648] HybridClock initialized: now 1781966777108870 us; error 25 us; skew 500 ppm
Jun 20 14:46:17 dist-test-slave-cx89 krb5kdc[6423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:17.111696 7120 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 14:46:17.112131 7120 webserver.cc:460] Webserver started at http://127.0.98.60:38743/ using document root <none> and password file <none>
I20260620 14:46:17.112323 7120 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:17.112368 7120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:17.112486 7120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:17.113480 7120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/data/instance:
uuid: "cc53f0865d504188a6310363ab89733b"
format_stamp: "Formatted at 2026-06-20 14:46:17 on dist-test-slave-cx89"
server_key: "41e4854fe83b81d1bf96d8ec68233a47"
server_key_iv: "08f834dac0580f570333489a9140c75b"
server_key_version: "encryptionkey@0"
I20260620 14:46:17.113811 7120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/wal/instance:
uuid: "cc53f0865d504188a6310363ab89733b"
format_stamp: "Formatted at 2026-06-20 14:46:17 on dist-test-slave-cx89"
server_key: "41e4854fe83b81d1bf96d8ec68233a47"
server_key_iv: "08f834dac0580f570333489a9140c75b"
server_key_version: "encryptionkey@0"
I20260620 14:46:17.114974 7120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:46:17.115612 7135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:17.115753 7120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:17.115831 7120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/wal
uuid: "cc53f0865d504188a6310363ab89733b"
format_stamp: "Formatted at 2026-06-20 14:46:17 on dist-test-slave-cx89"
server_key: "41e4854fe83b81d1bf96d8ec68233a47"
server_key_iv: "08f834dac0580f570333489a9140c75b"
server_key_version: "encryptionkey@0"
I20260620 14:46:17.115891 7120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:17.123056 7120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:17.125025 7120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:17.125149 7120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:17.129127 7120 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35679
I20260620 14:46:17.129153 7187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35679 every 8 connection(s)
I20260620 14:46:17.129463 7120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:46:17.130355 7188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:17.131841 7188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:46:17 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:46:17.138612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 7120
I20260620 14:46:17.140386 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.132563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37714 (local address 127.0.98.62:40459)
0620 14:46:17.132623 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.132625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.133108 (+ 483us) server_negotiation.cc:407] Connection header received
0620 14:46:17.133562 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.133563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.133576 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.133629 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.134201 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.134680 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.135689 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.135869 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.135920 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.137764 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.137766 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.137766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.137773 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.139010 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.139490 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.139492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.139492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.139550 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.139798 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.139799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.139799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.139893 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.139936 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.140217 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.140294 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:46:17.140444 7190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.132436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.0.1:37714)
0620 14:46:17.132579 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.132623 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.133473 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.133698 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.133701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.133929 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:17.134119 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.134124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.134765 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.134767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.135607 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.135609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.135928 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.135929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.136047 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.136473 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:17.136481 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:17.137685 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:17.139121 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.139126 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.139127 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.139385 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.139605 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.139606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.139606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.139741 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.139942 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:17.139947 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:17.140089 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.140169 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.140286 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260620 14:46:17.140908 7188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:46:17 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 14:46:17 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:46:17.146831 7191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.139085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51194 (local address 127.0.98.60:35679)
0620 14:46:17.139223 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.139227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.139240 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:46:17.139257 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.139258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.139268 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.139324 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.139987 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.140853 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.141720 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.142002 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.142052 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.144362 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.144367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.144368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.144373 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.145776 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.146115 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.146118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.146118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.146165 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.146327 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.146327 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.146328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.146488 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.146640 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.146687 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.146755 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260620 14:46:17.147262 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.141196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44906 (local address 127.0.98.61:35981)
0620 14:46:17.141235 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.141237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.141243 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:46:17.141265 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.141266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.141276 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.141334 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.141528 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.142061 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.142811 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.143021 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.143073 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.144774 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.144776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.144776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.144783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.146193 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.146480 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.146481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.146481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.146527 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.146689 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.146690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.146690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.146813 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.146871 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.147060 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.147168 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:46:17.147559 7190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.141126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35981 (local address 127.0.0.1:44906)
0620 14:46:17.141162 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.141171 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.141200 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.141337 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.141337 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.141431 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:17.141470 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.141471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.142145 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.142146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.142736 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.142738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.143075 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.143076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.143137 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.143463 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:17.143467 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:17.144690 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:17.146254 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.146255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.146255 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.146421 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.146583 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.146584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.146584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.146640 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.146898 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:17.146899 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:17.146943 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.147360 (+ 417us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.147479 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 14:46:17.148854 7188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35679 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:46:17 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:46:17.154029 7190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.149069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35679 (local address 127.0.0.1:51196)
0620 14:46:17.149147 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.149156 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.149192 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.149266 (+ 74us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.149267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.149363 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:17.149390 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.149391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.149910 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.149911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.150365 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.150366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.150704 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.150705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.150736 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.151081 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:17.151086 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:17.152124 (+ 1038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:17.153311 (+ 1187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.153312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.153312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.153468 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.153618 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.153618 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.153619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.153672 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.153816 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:17.153817 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:17.153845 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.153884 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.153948 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 14:46:17.154076 7191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.149112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51196 (local address 127.0.98.60:35679)
0620 14:46:17.149140 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.149141 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.149182 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:46:17.149223 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.149224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.149231 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.149263 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.149425 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.149865 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.150439 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.150645 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.150697 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.152188 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.152189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.152189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.152194 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.153249 (+ 1055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.153530 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.153531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.153531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.153574 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.153705 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.153705 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.153705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.153784 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.153823 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.153901 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.153988 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":14}
I20260620 14:46:17.155318 7188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: Bootstrap starting.
I20260620 14:46:17.156106 7188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:17.156324 7188 log.cc:824] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:17.156965 7188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: No bootstrap required, opened a new log
I20260620 14:46:17.158318 7188 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35679 } }
I20260620 14:46:17.158442 7188 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:17.158463 7188 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc53f0865d504188a6310363ab89733b, State: Initialized, Role: FOLLOWER
I20260620 14:46:17.158556 7188 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35679 } }
I20260620 14:46:17.158809 7188 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:46:17.158785 7193 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35679 } } }
I20260620 14:46:17.158864 7193 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:46:17.160288 7204 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:46:17.160341 7204 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:46:17.162365 7120 master_runner.cc:428] Detected that this master cc53f0865d504188a6310363ab89733b is joining an existing cluster
I20260620 14:46:17.162411 7120 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:35679
I20260620 14:46:17.184878 7191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.163893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51198 (local address 127.0.98.60:35679)
0620 14:46:17.163926 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.163927 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.164150 (+ 223us) server_negotiation.cc:407] Connection header received
0620 14:46:17.164225 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.164226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.164232 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.164256 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.165533 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.166206 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.167150 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.167406 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.167491 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.168320 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.168321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.168321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.168327 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.169458 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.183985 (+ 14527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.183987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.183987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.184048 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.184304 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.184305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.184305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.184431 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.184472 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.184673 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.184778 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:46:17.184878 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.163489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.0.98.62:40459)
0620 14:46:17.163548 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.163550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.164053 (+ 503us) server_negotiation.cc:407] Connection header received
0620 14:46:17.164138 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.164139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.164148 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.164176 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.164550 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.165005 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.166034 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.166320 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.166358 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.167334 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.167336 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.167336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.167344 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.168852 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.183993 (+ 15141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.183995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.183995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.184056 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.184320 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.184321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.184322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.184442 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.184481 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.184706 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.184778 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:46:17.185200 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.163690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.0.1:37726)
0620 14:46:17.163990 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.163998 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.164076 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.164231 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.164232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.164433 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:17.164484 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.164485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.165069 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.165070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.165931 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.165933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.166440 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.166440 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.166496 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.166894 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:17.166899 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:17.167254 (+ 355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:17.168928 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.168929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.168929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.183880 (+ 14951us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.184127 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.184128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.184128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.184251 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.184551 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:17.184551 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:17.184591 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.184995 (+ 404us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.185106 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":55,"threads_started":1}
I20260620 14:46:17.185635 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.163857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35679 (local address 127.0.0.1:51198)
0620 14:46:17.164102 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.164111 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.164169 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.164300 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.164301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.165392 (+ 1091us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:17.165446 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.165449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.166288 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.166288 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.167073 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.167076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.167492 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.167492 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.167559 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.167928 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:17.167932 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:17.168260 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:17.169522 (+ 1262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.169523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.169523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.183880 (+ 14357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.184127 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.184128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.184128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.184239 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.184511 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:17.184511 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:17.184575 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.185447 (+ 872us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.185534 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":90,"threads_started":1}
I20260620 14:46:17.187759 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.163400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44918 (local address 127.0.98.61:35981)
0620 14:46:17.163455 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.163456 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.163862 (+ 406us) server_negotiation.cc:407] Connection header received
0620 14:46:17.163967 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.163968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.163979 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.164033 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.164404 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.165155 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.166079 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.166358 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.166392 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.185027 (+ 18635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.185029 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.185029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.185037 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.186520 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.186804 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.186805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.186805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.186859 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.187140 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.187141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.187141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.187259 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.187319 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.187604 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.187679 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:46:17.187776 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.163343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35981 (local address 127.0.0.1:44918)
0620 14:46:17.163805 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.163815 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.163887 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.164050 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.164051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.164231 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:17.164317 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.164318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.165224 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.165225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.166004 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.166005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.166453 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.166453 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.166504 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.184411 (+ 17907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:17.184416 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:17.184912 (+ 496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:17.186595 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.186595 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.186595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.186732 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.186924 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.186924 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.186925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.186986 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.187405 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:17.187406 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:17.187470 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.187569 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.187669 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":106,"threads_started":1}
Jun 20 14:46:17 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:46:17.193089 7191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.187353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51210 (local address 127.0.98.60:35679)
0620 14:46:17.187395 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.187397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.187402 (+ 5us) server_negotiation.cc:407] Connection header received
0620 14:46:17.187446 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.187447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.187456 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.187501 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.187816 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.188389 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.189024 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.189211 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.189261 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.191057 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.191058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.191058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.191063 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.192205 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.192542 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.192542 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.192542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.192586 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.192740 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.192740 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.192741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.192818 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.192857 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.192987 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.193033 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:46:17.193156 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.187260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35679 (local address 127.0.0.1:51210)
0620 14:46:17.187309 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.187318 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.187349 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.187527 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.187529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.187690 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:17.187736 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.187737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.188462 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.188463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.188960 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.188962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.189264 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.189265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.189325 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.189689 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:17.189692 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:17.190979 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:17.192272 (+ 1293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.192274 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.192274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.192474 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.192633 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.192634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.192634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.192690 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.192863 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:17.192863 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:17.192906 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.192971 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.193061 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:46:17.193601 6468 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:35679: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cc53f0865d504188a6310363ab89733b"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 35679
}
attrs {
promote: true
}
}
dest_uuid: "d08956433a1142789be562fbf8eb6245"
cas_config_opid_index: 5
I20260620 14:46:17.193815 6468 consensus_queue.cc:237] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: true } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } }
I20260620 14:46:17.194114 7028 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cc53f0865d504188a6310363ab89733b (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: true } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } } }
I20260620 14:46:17.195050 7028 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:35679
I20260620 14:46:17.195137 7027 sys_catalog.cc:455] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d08956433a1142789be562fbf8eb6245" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: true } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } } }
I20260620 14:46:17.195214 7027 sys_catalog.cc:458] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:17.195292 7120 master.cc:537] Master@127.0.98.60:35679 shutting down...
I20260620 14:46:17.195284 7162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:17.196265 7162 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b [term 1 FOLLOWER]: Refusing update from remote peer d08956433a1142789be562fbf8eb6245: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:46:17.196557 7027 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:46:17.197180 6455 consensus_peers.cc:487] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 -> Peer cc53f0865d504188a6310363ab89733b (127.0.98.60:35679): Couldn't send request to peer cc53f0865d504188a6310363ab89733b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:46:17.198596 7120 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:46:17.198679 7120 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:17.198719 7120 tablet_replica.cc:330] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: stopping tablet replica
I20260620 14:46:17.198969 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.194166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35981 (local address 127.0.0.1:44926)
0620 14:46:17.195562 (+ 1396us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.195572 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.195619 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.195823 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.195824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.195825 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:46:17.195859 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.195860 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.196742 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.196743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.197846 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.197849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.198740 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.198741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.198801 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.198823 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.198871 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.198908 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1383}
I20260620 14:46:17.199165 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.195382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44926 (local address 127.0.98.61:35981)
0620 14:46:17.195534 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.195536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.195664 (+ 128us) server_negotiation.cc:407] Connection header received
0620 14:46:17.195684 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.195685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.195705 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.195776 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:46:17.195920 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.196664 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.197935 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.198557 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.198628 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.198708 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.199075 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.199112 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260620 14:46:17.199491 7076 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 LEARNER]: Refusing update from remote peer d08956433a1142789be562fbf8eb6245: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:46:17.199667 7027 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 14:46:17.200294 7076 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cc53f0865d504188a6310363ab89733b (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: true } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } } }
I20260620 14:46:17.201390 7106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d08956433a1142789be562fbf8eb6245. Latest consensus state: current_term: 1 leader_uuid: "d08956433a1142789be562fbf8eb6245" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: true } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } } }
I20260620 14:46:17.201458 7106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [sys.catalog]: This master's current role is: LEARNER
I20260620 14:46:17.201484 7106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d08956433a1142789be562fbf8eb6245" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: true } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } } }
I20260620 14:46:17.201524 7106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [sys.catalog]: This master's current role is: LEARNER
I20260620 14:46:17.201777 7027 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245: attempting to promote NON_VOTER 6604b93b1d5b4df4963aea5f8e50a62a to VOTER
I20260620 14:46:17.201944 7027 consensus_queue.cc:237] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: false } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } }
I20260620 14:46:17.202328 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.202248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35679 (local address 127.0.0.1:51212)
0620 14:46:17.202278 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.202309 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 14:46:17.202453 7076 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 LEARNER]: Refusing update from remote peer d08956433a1142789be562fbf8eb6245: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 14:46:17.202610 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.202524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35679 (local address 127.0.0.1:51226)
0620 14:46:17.202559 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.202589 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 14:46:17.202638 7215 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260620 14:46:17.202730 6455 consensus_peers.cc:487] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 -> Peer cc53f0865d504188a6310363ab89733b (127.0.98.60:35679): Couldn't send request to peer cc53f0865d504188a6310363ab89733b. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 14:46:17.203233 7027 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6604b93b1d5b4df4963aea5f8e50a62a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: false } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } } }
I20260620 14:46:17.203406 7076 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6604b93b1d5b4df4963aea5f8e50a62a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: false } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } } }
I20260620 14:46:17.203980 7027 sys_catalog.cc:455] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d08956433a1142789be562fbf8eb6245" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: false } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } } }
I20260620 14:46:17.204052 7027 sys_catalog.cc:458] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:17.204111 7027 sys_catalog.cc:455] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d08956433a1142789be562fbf8eb6245" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: false } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } } }
I20260620 14:46:17.204147 7027 sys_catalog.cc:458] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:17.204227 7214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d08956433a1142789be562fbf8eb6245" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: false } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } } }
I20260620 14:46:17.204299 7214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:46:17.211496 7120 master.cc:552] Master@127.0.98.60:35679 shutdown complete.
I20260620 14:46:17.211551 7120 master_runner.cc:305] Clearing existing system tablet
I20260620 14:46:17.211674 7120 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:17.211712 7120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:17.212215 7120 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:17.212620 7217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:17.212733 7120 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:17.212800 7120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:17.212823 7120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/wal
uuid: "cc53f0865d504188a6310363ab89733b"
format_stamp: "Formatted at 2026-06-20 14:46:17 on dist-test-slave-cx89"
server_key: "41e4854fe83b81d1bf96d8ec68233a47"
server_key_iv: "08f834dac0580f570333489a9140c75b"
server_key_version: "encryptionkey@0"
I20260620 14:46:17.213532 7120 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:17.214798 7120 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:46:17.214862 7120 log.cc:1196] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 14:46:17.215040 7120 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: Deleting consensus metadata
I20260620 14:46:17.215163 7120 master_runner.cc:315] Copying system tablet from 127.0.98.62:40459
I20260620 14:46:17.215782 7120 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40459
I20260620 14:46:17.221690 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.216107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37732 (local address 127.0.98.62:40459)
0620 14:46:17.216149 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.216151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.216178 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:46:17.216249 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.216250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.216258 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.216300 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.216624 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.217087 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.218150 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.218443 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.218505 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.219507 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.219509 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.219509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.219516 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.220755 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.221108 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.221108 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.221109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.221154 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.221301 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.221301 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.221301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.221388 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.221431 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.221576 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.221628 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:46:17.221784 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.216020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.0.1:37732)
0620 14:46:17.216118 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.216129 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.216193 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.216316 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.216317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.216512 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:17.216566 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.216568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.217149 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.217150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.218055 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.218058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.218516 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.218517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.218577 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.219014 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:17.219019 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:17.219384 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:17.220819 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.220821 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.220821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.221038 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.221193 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.221193 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.221194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.221255 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.221435 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:17.221437 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:17.221483 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.221553 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.221663 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":56,"threads_started":1}
I20260620 14:46:17.221984 6488 tablet_copy_service.cc:140] P d08956433a1142789be562fbf8eb6245: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cc53f0865d504188a6310363ab89733b ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:37732)
I20260620 14:46:17.222031 6488 tablet_copy_service.cc:161] P d08956433a1142789be562fbf8eb6245: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cc53f0865d504188a6310363ab89733b at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:37732: session id = cc53f0865d504188a6310363ab89733b-00000000000000000000000000000000
I20260620 14:46:17.222427 6488 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:46:17.222870 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:46:17.224149 7120 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: tablet copy: Starting download of 0 data blocks...
I20260620 14:46:17.224289 7120 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: tablet copy: Starting download of 1 WAL segments...
I20260620 14:46:17.224951 7120 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:46:17.225831 6488 tablet_copy_service.cc:342] P d08956433a1142789be562fbf8eb6245: Request end of tablet copy session cc53f0865d504188a6310363ab89733b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:37732
I20260620 14:46:17.225886 6488 tablet_copy_service.cc:433] P d08956433a1142789be562fbf8eb6245: ending tablet copy session cc53f0865d504188a6310363ab89733b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cc53f0865d504188a6310363ab89733b
I20260620 14:46:17.227972 7120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:17.229064 7224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:17.229099 7227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:17.229305 7120 server_base.cc:1034] running on GCE node
W20260620 14:46:17.235024 7225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:17.235181 7120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:17.235328 7120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:17.236441 7120 hybrid_clock.cc:648] HybridClock initialized: now 1781966777236436 us; error 27 us; skew 500 ppm
Jun 20 14:46:17 dist-test-slave-cx89 krb5kdc[6423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:17.237885 7120 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 14:46:17.238152 7120 webserver.cc:460] Webserver started at http://127.0.98.60:34787/ using document root <none> and password file <none>
I20260620 14:46:17.238269 7120 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:17.238307 7120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:17.239017 7120 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:17.239632 7234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:17.239750 7120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:17.239792 7120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/wal
uuid: "cc53f0865d504188a6310363ab89733b"
format_stamp: "Formatted at 2026-06-20 14:46:17 on dist-test-slave-cx89"
server_key: "41e4854fe83b81d1bf96d8ec68233a47"
server_key_iv: "08f834dac0580f570333489a9140c75b"
server_key_version: "encryptionkey@0"
I20260620 14:46:17.239845 7120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:17.262367 7120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:17.263865 7120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:17.267103 7120 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35679
I20260620 14:46:17.267202 7286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35679 every 8 connection(s)
I20260620 14:46:17.267462 7120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:46:17.268119 7287 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:46:17.268491 7287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: Bootstrap starting.
I20260620 14:46:17.270561 7287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:46:17.270721 7287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: Bootstrap complete.
I20260620 14:46:17.270913 7287 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: false } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } }
I20260620 14:46:17.270983 7287 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc53f0865d504188a6310363ab89733b, State: Initialized, Role: LEARNER
I20260620 14:46:17.271008 7287 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: false } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } }
I20260620 14:46:17.271166 7287 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:46:17.271178 7290 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: false } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } } }
I20260620 14:46:17.271277 7290 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b [sys.catalog]: This master's current role is: LEARNER
I20260620 14:46:17.272482 7301 catalog_manager.cc:1259] Loaded cluster ID: 61a4b4a0e1f64df3b241f2a80d00c0c9
I20260620 14:46:17.272527 7301 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: loading cluster ID for follower catalog manager: success
I20260620 14:46:17.273554 7301 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: acquiring CA information for follower catalog manager: success
I20260620 14:46:17.273811 7301 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cc53f0865d504188a6310363ab89733b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:46:17.306420 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.301059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.0.98.60:35679)
0620 14:46:17.301163 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.301165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.301173 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:46:17.301196 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.301197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.301206 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.301269 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.301763 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.302386 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.303080 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.303272 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.303323 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.304485 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.304486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.304486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.304492 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.305582 (+ 1090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.305889 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.305890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.305890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.305932 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.306059 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.306059 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.306060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.306152 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.306217 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.306297 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.306352 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":55,"threads_started":1}
I20260620 14:46:17.306869 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 6530
I20260620 14:46:17.310979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38171
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40459
--webserver_port=44473
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40459,127.0.98.61:35981
--tserver_master_addrs=127.0.98.62:40459,127.0.98.61:35981,127.0.98.60:35679 with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:17.389058 7303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:17.389259 7303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:17.389290 7303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:17.390367 7303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:17.390406 7303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:17.390440 7303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:46:17.391908 7303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44473
--tserver_master_addrs=127.0.98.62:40459,127.0.98.61:35981,127.0.98.60:35679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:17.392127 7303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:17.392316 7303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:17.396914 7308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:17.396876 7309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:17.396876 7311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:17.397047 7303 server_base.cc:1034] running on GCE node
I20260620 14:46:17.397378 7303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:17.397612 7303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:17.398747 7303 hybrid_clock.cc:648] HybridClock initialized: now 1781966777398736 us; error 30 us; skew 500 ppm
Jun 20 14:46:17 dist-test-slave-cx89 krb5kdc[6423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:17.401044 7303 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:46:17.401450 7303 webserver.cc:460] Webserver started at http://127.0.98.1:44473/ using document root <none> and password file <none>
I20260620 14:46:17.401645 7303 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:17.401705 7303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:17.402943 7303 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:17.403491 7318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:17.403628 7303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:17.403707 7303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "7f50bab07914483c9672fb1b82ea9a3f"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "fce99534ae70f5e9599d4557b8d35bbe"
server_key_iv: "485a724a0976f7a5371f3754f51fcf2b"
server_key_version: "encryptionkey@0"
I20260620 14:46:17.404016 7303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:17.414562 7303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:17.416437 7303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:17.416560 7303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:17.416760 7303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:17.417019 7303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:17.417050 7303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:17.417078 7303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:17.417095 7303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:17.422407 7303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38171
I20260620 14:46:17.422459 7431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38171 every 8 connection(s)
I20260620 14:46:17.422760 7303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:46:17.425047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 7303
I20260620 14:46:17.425195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 6664
I20260620 14:46:17.430804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40311
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40459
--webserver_port=36821
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40459,127.0.98.61:35981
--tserver_master_addrs=127.0.98.62:40459,127.0.98.61:35981,127.0.98.60:35679 with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:46:17 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 14:46:17 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 14:46:17 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:46:17.435307 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.423379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40819 (local address 127.0.98.60:35679)
0620 14:46:17.423476 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.423478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.423977 (+ 499us) server_negotiation.cc:407] Connection header received
0620 14:46:17.424436 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.424437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.424450 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.424509 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.426053 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.426771 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.428395 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.428725 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.428767 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.431566 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.431568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.431568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.431575 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.433245 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.434283 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.434284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.434285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.434335 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.434589 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.434590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.434590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.434713 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.434776 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.435128 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.435215 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 14:46:17.435461 7437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.423692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:47135)
0620 14:46:17.424502 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.424511 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.424575 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.424718 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.424719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.424988 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:17.425800 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.425807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.426818 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.426819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.429534 (+ 2715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.429537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.430110 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.430111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.430174 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.430950 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:17.430954 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:17.432280 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:17.434197 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.434198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.434199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.434397 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.434585 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.434585 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.434586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.434637 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.434900 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:17.434900 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:17.434968 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.435070 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.435222 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":1833,"thread_start_us":55,"threads_started":1}
I20260620 14:46:17.435802 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.423397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35981 (local address 127.0.98.1:43285)
0620 14:46:17.423812 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.423849 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.424340 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.424597 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.424601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.424979 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:17.428844 (+ 3865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.428845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.429714 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.429715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.430432 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.430434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.431470 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.431471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.431536 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.431886 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:17.431891 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:17.433295 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:17.434756 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.434757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.434758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.434937 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.435142 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.435143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.435143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.435199 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.435518 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:17.435519 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:17.435561 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.435613 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.435700 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":26,"spinlock_wait_cycles":6912,"thread_start_us":82,"threads_started":1}
I20260620 14:46:17.435823 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.423767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47135 (local address 127.0.98.62:40459)
0620 14:46:17.423820 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.423822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.424554 (+ 732us) server_negotiation.cc:407] Connection header received
0620 14:46:17.424630 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.424632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.424644 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.424697 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.425911 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.426697 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.429638 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.429916 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.429957 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.432364 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.432365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.432366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.432373 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.434111 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.434462 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.434463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.434463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.434510 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.434696 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.434697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.434697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.434829 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.434879 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.435666 (+ 787us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.435741 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:46:17.435930 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.423397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35679 (local address 127.0.98.1:40819)
0620 14:46:17.423802 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.423849 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.424351 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.424596 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.424600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.425922 (+ 1322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:17.425969 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.425971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.426851 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.426851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.428281 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.428283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.428846 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.428847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.428979 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.429509 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:17.429519 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:17.431470 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:17.433324 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.433681 (+ 357us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.433686 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.434203 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.434398 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.434399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.434399 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.434518 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.434804 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:17.434810 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:17.434968 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.435713 (+ 745us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.435803 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":41,"thread_start_us":76,"threads_started":1}
I20260620 14:46:17.436199 7433 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40459
I20260620 14:46:17.436292 7433 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:17.436393 7432 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35679
I20260620 14:46:17.436447 7432 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:17.436460 7433 heartbeater.cc:511] Master 127.0.98.62:40459 requested a full tablet report, sending...
I20260620 14:46:17.436515 7432 heartbeater.cc:511] Master 127.0.98.60:35679 requested a full tablet report, sending...
I20260620 14:46:17.436612 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.423372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43285 (local address 127.0.98.61:35981)
0620 14:46:17.423436 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.423438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.423977 (+ 539us) server_negotiation.cc:407] Connection header received
0620 14:46:17.424433 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.424434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.424447 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.424513 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.428928 (+ 4415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.429617 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.430519 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.430814 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.430879 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.433373 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.433374 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.433375 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.433381 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.434662 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.435014 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.435015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.435015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.435070 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.435260 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.435261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.435261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.435390 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.435501 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.436458 (+ 957us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.436534 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:46:17.436834 6468 ts_manager.cc:194] Re-registered known tserver with Master: 7f50bab07914483c9672fb1b82ea9a3f (127.0.98.1:38171)
I20260620 14:46:17.436939 7251 ts_manager.cc:194] Registered new tserver with Master: 7f50bab07914483c9672fb1b82ea9a3f (127.0.98.1:38171)
I20260620 14:46:17.437096 7434 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35981
I20260620 14:46:17.437152 7434 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:17.437211 7434 heartbeater.cc:511] Master 127.0.98.61:35981 requested a full tablet report, sending...
I20260620 14:46:17.437290 6468 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47135
I20260620 14:46:17.437577 7066 ts_manager.cc:194] Registered new tserver with Master: 7f50bab07914483c9672fb1b82ea9a3f (127.0.98.1:38171)
W20260620 14:46:17.512902 7439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:17.513092 7439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:17.513123 7439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:17.514205 7439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:17.514254 7439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:17.514288 7439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:46:17.515762 7439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36821
--tserver_master_addrs=127.0.98.62:40459,127.0.98.61:35981,127.0.98.60:35679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:17.515940 7439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:17.516134 7439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:17.520710 7445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:17.520718 7447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:17.520829 7444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:17.520720 7439 server_base.cc:1034] running on GCE node
I20260620 14:46:17.521054 7439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:17.521312 7439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:17.522444 7439 hybrid_clock.cc:648] HybridClock initialized: now 1781966777522431 us; error 27 us; skew 500 ppm
Jun 20 14:46:17 dist-test-slave-cx89 krb5kdc[6423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:17.524696 7439 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:46:17.525069 7439 webserver.cc:460] Webserver started at http://127.0.98.2:36821/ using document root <none> and password file <none>
I20260620 14:46:17.525266 7439 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:17.525323 7439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:17.526525 7439 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:17.526994 7454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:17.527127 7439 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:17.527204 7439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "e07ea2005fa84020a75becdaa5468e7d"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "de37e2d42de54c438b732f8804eb44ce"
server_key_iv: "7305367bf795968bee38ea50ef971a8e"
server_key_version: "encryptionkey@0"
I20260620 14:46:17.527550 7439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:17.554057 7439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:17.556136 7439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:17.556262 7439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:17.556468 7439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:17.556756 7439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:17.556793 7439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:17.556823 7439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:17.556843 7439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:17.561911 7439 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40311
I20260620 14:46:17.561975 7567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40311 every 8 connection(s)
I20260620 14:46:17.562269 7439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:46:17.566689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 7439
I20260620 14:46:17.566818 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 6798
Jun 20 14:46:17 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 14:46:17 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:46:17.572893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40747
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40459
--webserver_port=43397
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40459,127.0.98.61:35981
--tserver_master_addrs=127.0.98.62:40459,127.0.98.61:35981,127.0.98.60:35679 with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:46:17.574198 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.562916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46885 (local address 127.0.98.62:40459)
0620 14:46:17.562975 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.562977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.563235 (+ 258us) server_negotiation.cc:407] Connection header received
0620 14:46:17.564311 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.564312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.564323 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.564353 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.565110 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.565831 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.568517 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.568827 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.568880 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.571223 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.571224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.571224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.571232 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.572798 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.573263 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.573265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.573265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.573327 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.573588 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.573589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.573590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.573703 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.573758 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.574023 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.574113 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:46:17.574360 7573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.562926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:46885)
0620 14:46:17.563181 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.563191 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.564204 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.564428 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.564432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.564840 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:17.565001 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.565007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.565909 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.565910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.566860 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.566863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.568930 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.568931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.569033 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.569564 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:17.569575 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:17.571111 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:17.572882 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.572890 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.572892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.573182 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.573394 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.573395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.573395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.573524 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.573758 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:17.573764 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:17.573900 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.573989 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.574145 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":10,"thread_start_us":106,"threads_started":1}
Jun 20 14:46:17 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:46:17.574862 7569 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40459
I20260620 14:46:17.574961 7569 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:17.575143 7569 heartbeater.cc:511] Master 127.0.98.62:40459 requested a full tablet report, sending...
I20260620 14:46:17.575730 6468 ts_manager.cc:194] Re-registered known tserver with Master: e07ea2005fa84020a75becdaa5468e7d (127.0.98.2:40311)
I20260620 14:46:17.576071 6468 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46885
I20260620 14:46:17.576478 7574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.563198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35679 (local address 127.0.98.2:38069)
0620 14:46:17.563327 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.563335 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.564201 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.564428 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.564432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.564811 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:17.565000 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.565004 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.567628 (+ 2624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.567629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.568395 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.568398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.568889 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.568890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.569033 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.569609 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:17.569614 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:17.571367 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:17.572933 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.572933 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.572934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.573362 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.573533 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.573534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.573534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.573596 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.576178 (+ 2582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:17.576179 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:17.576221 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.576286 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.576384 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
I20260620 14:46:17.576508 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.563269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38069 (local address 127.0.98.60:35679)
0620 14:46:17.563308 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.563310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.563374 (+ 64us) server_negotiation.cc:407] Connection header received
0620 14:46:17.564275 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.564276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.564287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.564338 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.565080 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.565836 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.568483 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.568795 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.568865 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.571461 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.571462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.571463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.571471 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.572840 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.573430 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.573431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.573432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.573484 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.575489 (+ 2005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.575490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.575491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.575612 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.575653 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.576338 (+ 685us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.576419 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:46:17.577072 7568 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35679
I20260620 14:46:17.577373 7568 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:17.577466 7568 heartbeater.cc:511] Master 127.0.98.60:35679 requested a full tablet report, sending...
I20260620 14:46:17.577996 7251 ts_manager.cc:194] Registered new tserver with Master: e07ea2005fa84020a75becdaa5468e7d (127.0.98.2:40311)
I20260620 14:46:17.579231 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.562989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33917 (local address 127.0.98.61:35981)
0620 14:46:17.563036 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.563038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.563229 (+ 191us) server_negotiation.cc:407] Connection header received
0620 14:46:17.564311 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.564312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.564323 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.564353 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.565108 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.565584 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.572713 (+ 7129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.573008 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.573071 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.576912 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.576913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.576913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.576922 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.578285 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.578580 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.578581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.578582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.578631 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.578788 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.578789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.578789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.578890 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.578941 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.579075 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.579145 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:46:17.579252 7572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.562925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35981 (local address 127.0.98.2:33917)
0620 14:46:17.563106 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.563146 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.564192 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.564428 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.564432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.564848 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:17.565001 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.565007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.565674 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.565675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.566860 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.566863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.573154 (+ 6291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.573155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.573224 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.573687 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:17.573692 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:17.576809 (+ 3117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:17.578359 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.578360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.578361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.578509 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.578679 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.578680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.578680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.578732 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.578939 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:17.578940 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:17.578981 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.579034 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.579145 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":6,"thread_start_us":103,"threads_started":1}
I20260620 14:46:17.579594 7570 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35981
I20260620 14:46:17.579643 7570 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:17.579682 7570 heartbeater.cc:511] Master 127.0.98.61:35981 requested a full tablet report, sending...
I20260620 14:46:17.579921 7066 ts_manager.cc:194] Registered new tserver with Master: e07ea2005fa84020a75becdaa5468e7d (127.0.98.2:40311)
W20260620 14:46:17.652787 7575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:17.652987 7575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:17.653017 7575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:17.654172 7575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:17.654214 7575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:17.654250 7575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:46:17.655705 7575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43397
--tserver_master_addrs=127.0.98.62:40459,127.0.98.61:35981,127.0.98.60:35679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:17.655934 7575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:17.656139 7575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:17.660954 7583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:17.660950 7581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:17.660946 7580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:17.661146 7575 server_base.cc:1034] running on GCE node
I20260620 14:46:17.661309 7575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:17.661511 7575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:17.662650 7575 hybrid_clock.cc:648] HybridClock initialized: now 1781966777662637 us; error 30 us; skew 500 ppm
Jun 20 14:46:17 dist-test-slave-cx89 krb5kdc[6423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:17.664952 7575 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:46:17.665345 7575 webserver.cc:460] Webserver started at http://127.0.98.3:43397/ using document root <none> and password file <none>
I20260620 14:46:17.665539 7575 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:17.665591 7575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:17.666834 7575 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:17.667315 7590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:17.667464 7575 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:17.667542 7575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "a7332d57250a4c4da5689a2dec888ff3"
format_stamp: "Formatted at 2026-06-20 14:46:16 on dist-test-slave-cx89"
server_key: "655212978fd79628017e58672dd47786"
server_key_iv: "fc0e6560a7f640835304b6ed98fbafa8"
server_key_version: "encryptionkey@0"
I20260620 14:46:17.667861 7575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:17.682663 7575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:17.684762 7575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:17.684894 7575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:17.685098 7575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:17.685377 7575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:17.685408 7575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:17.685437 7575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:17.685460 7575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:17.690620 7575 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40747
I20260620 14:46:17.690685 7703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40747 every 8 connection(s)
I20260620 14:46:17.690989 7575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:46:17 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 14:46:17 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 14:46:17 dist-test-slave-cx89 krb5kdc[6423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:46:17.699189 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 7575
I20260620 14:46:17.700765 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.692018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57257 (local address 127.0.98.60:35679)
0620 14:46:17.692067 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.692069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.692128 (+ 59us) server_negotiation.cc:407] Connection header received
0620 14:46:17.692720 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.692722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.692731 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.692783 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.693486 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.694224 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.695089 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.695353 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.695431 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.698050 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.698051 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.698052 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.698059 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.699477 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.699808 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.699809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.699809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.699859 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.700068 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.700069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.700069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.700193 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.700234 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.700605 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.700669 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:46:17.700765 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.691645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45417 (local address 127.0.98.62:40459)
0620 14:46:17.691696 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.691698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.691931 (+ 233us) server_negotiation.cc:407] Connection header received
0620 14:46:17.692604 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.692605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.692617 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.692670 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.693481 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.693973 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.695163 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.695458 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.695512 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.697647 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.697649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.697649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.697655 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.699160 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.699757 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.699758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.699759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.699815 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.700068 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.700069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.700069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.700193 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.700234 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.700591 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.700670 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:46:17.701299 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.692031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38843 (local address 127.0.98.61:35981)
0620 14:46:17.692072 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.692074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.692082 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:46:17.692476 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.692477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.692489 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.692537 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.693458 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.694216 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.695219 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.695404 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.695479 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.698606 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.698607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.698608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.698615 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.700074 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.700444 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.700445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.700445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.700503 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.700727 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.700728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.700729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.700858 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.700901 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.701136 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.701212 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:46:17.701300 7710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.691920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35679 (local address 127.0.98.3:57257)
0620 14:46:17.692076 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.692084 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.692399 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.692826 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.692826 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.693168 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:17.693370 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.693376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.694319 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.694320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.695009 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.695011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.695779 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.695779 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.695818 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.696260 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:17.696265 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:17.697979 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:17.699551 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.699552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.699552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.699739 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.699924 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.699925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.699925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.699998 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.700279 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:17.700285 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:17.700443 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.700875 (+ 432us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.701062 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":67,"thread_start_us":56,"threads_started":1}
I20260620 14:46:17.701300 7708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.691658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:45417)
0620 14:46:17.691803 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.691843 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.692382 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.692715 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.692719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.693165 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:17.693370 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.693376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.694065 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.694066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.695084 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.695086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.695513 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.695514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.695607 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.696151 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:17.696156 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:17.697560 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:17.699374 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.699381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.699383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.699672 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.699882 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.699883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.699883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.699999 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.700279 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:17.700285 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:17.700443 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.700874 (+ 431us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.701125 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":106,"thread_start_us":71,"threads_started":1}
I20260620 14:46:17.701748 7704 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35679
I20260620 14:46:17.701859 7704 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:17.701881 7709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.691658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35981 (local address 127.0.98.3:38843)
0620 14:46:17.691810 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.691844 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.692390 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.692925 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.692926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.693164 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:17.693370 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.693374 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.694292 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.694293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.695165 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.695166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.695485 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.695486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.695606 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.696091 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:17.696098 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:17.698490 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:17.700163 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.700164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.700164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.700361 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.700590 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:17.700591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:17.700591 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:17.700658 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:17.700994 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:17.700995 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:17.701036 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.701415 (+ 379us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.701536 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":42,"thread_start_us":79,"threads_started":1}
I20260620 14:46:17.701972 7705 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40459
I20260620 14:46:17.702001 7705 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:17.702044 7704 heartbeater.cc:511] Master 127.0.98.60:35679 requested a full tablet report, sending...
I20260620 14:46:17.702046 7705 heartbeater.cc:511] Master 127.0.98.62:40459 requested a full tablet report, sending...
I20260620 14:46:17.702167 7706 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35981
I20260620 14:46:17.702210 7706 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:17.702268 7706 heartbeater.cc:511] Master 127.0.98.61:35981 requested a full tablet report, sending...
I20260620 14:46:17.702406 6468 ts_manager.cc:194] Re-registered known tserver with Master: a7332d57250a4c4da5689a2dec888ff3 (127.0.98.3:40747)
I20260620 14:46:17.702457 7251 ts_manager.cc:194] Registered new tserver with Master: a7332d57250a4c4da5689a2dec888ff3 (127.0.98.3:40747)
I20260620 14:46:17.702587 7066 ts_manager.cc:194] Registered new tserver with Master: a7332d57250a4c4da5689a2dec888ff3 (127.0.98.3:40747)
I20260620 14:46:17.702852 6468 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45417
I20260620 14:46:17.709020 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.702427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44936 (local address 127.0.98.61:35981)
0620 14:46:17.702849 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.702850 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.702987 (+ 137us) server_negotiation.cc:407] Connection header received
0620 14:46:17.703075 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.703076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.703085 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.703112 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.703783 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.704481 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.705157 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.705350 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.705401 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.706700 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.706702 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.706702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.706708 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.708019 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.708391 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.708392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.708392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.708440 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.708598 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.708599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.708599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.708699 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.708781 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.708869 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.708936 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402}
I20260620 14:46:17.709719 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.701204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51262 (local address 127.0.98.60:35679)
0620 14:46:17.701240 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.701241 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.702957 (+ 1716us) server_negotiation.cc:407] Connection header received
0620 14:46:17.703042 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.703043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.703052 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.703077 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.703733 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.704442 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.705612 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.705900 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.705954 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.707338 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.707339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.707339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.707345 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.708732 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.709110 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.709111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.709111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.709161 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.709338 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.709338 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.709338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.709417 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.709474 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.709606 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.709655 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:46:17.710744 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.700509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.0.98.62:40459)
0620 14:46:17.700846 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.700848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.702790 (+ 1942us) server_negotiation.cc:407] Connection header received
0620 14:46:17.702881 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.702882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.702893 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.702920 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.703726 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.704323 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.706051 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.706341 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.706410 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.708150 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.708151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.708152 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.708158 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.709794 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.710127 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.710128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.710128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.710184 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.710331 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.710331 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.710332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.710427 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.710506 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.710601 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.710663 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":87,"threads_started":1}
I20260620 14:46:17.712114 6468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:46:17.713555 6468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:46:17.720178 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.716776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40747 (local address 127.0.0.1:48372)
0620 14:46:17.716940 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.716950 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.716999 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.717311 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.717312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.717314 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:46:17.717357 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.717359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.718242 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.718243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.719066 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.719068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.719975 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.719975 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.720026 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.720030 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.720066 (+ 36us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.720105 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260620 14:46:17.720238 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.716858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48372 (local address 127.0.98.3:40747)
0620 14:46:17.717023 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.717028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.717037 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:46:17.717090 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.717092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.717109 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.717184 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:46:17.717430 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.718165 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.719157 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.719911 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.719976 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.720109 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.720136 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.720174 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":75,"threads_started":1}
I20260620 14:46:17.720811 7638 tablet_service.cc:1467] Processing CreateTablet for tablet a841ddb651544210b74b5ab3985ebb80 (DEFAULT_TABLE table=test-table [id=737c1119a9d94d479c6f08eac679ebfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:17.720979 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.716819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38171 (local address 127.0.0.1:56598)
0620 14:46:17.717067 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.717074 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.717106 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.717405 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.717406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.717408 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:46:17.717468 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.717471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.718709 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.718710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.719855 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.719857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.720756 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.720757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.720825 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.720829 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.720885 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.720918 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":26,"thread_start_us":84,"threads_started":1}
I20260620 14:46:17.721015 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.717029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56598 (local address 127.0.98.1:38171)
0620 14:46:17.717196 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.717201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.717212 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:46:17.717250 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.717252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.717268 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.717343 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:46:17.717549 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.718624 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.719928 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.720664 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.720748 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.720823 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.720897 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.720939 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
I20260620 14:46:17.721169 7638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a841ddb651544210b74b5ab3985ebb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:17.721467 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.716825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40311 (local address 127.0.0.1:57374)
0620 14:46:17.717251 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.717260 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.717355 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.717576 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.717576 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.717578 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:46:17.717608 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.717609 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.718790 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.718791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.719753 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.719755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.720935 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.720936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.721302 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.721308 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.721364 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.721405 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":82,"threads_started":1}
I20260620 14:46:17.721642 7366 tablet_service.cc:1467] Processing CreateTablet for tablet a841ddb651544210b74b5ab3985ebb80 (DEFAULT_TABLE table=test-table [id=737c1119a9d94d479c6f08eac679ebfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:17.721777 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.717040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.0.98.2:40311)
0620 14:46:17.717204 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.717208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.717329 (+ 121us) server_negotiation.cc:407] Connection header received
0620 14:46:17.717433 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.717435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.717452 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.717498 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:46:17.717697 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.718717 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.719839 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.720783 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.720840 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.720904 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.721390 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.721430 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":78,"threads_started":1}
I20260620 14:46:17.721951 7366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a841ddb651544210b74b5ab3985ebb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:17.722702 7502 tablet_service.cc:1467] Processing CreateTablet for tablet a841ddb651544210b74b5ab3985ebb80 (DEFAULT_TABLE table=test-table [id=737c1119a9d94d479c6f08eac679ebfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:17.722935 7502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a841ddb651544210b74b5ab3985ebb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:17.724821 7731 tablet_bootstrap.cc:492] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3: Bootstrap starting.
I20260620 14:46:17.725343 7732 tablet_bootstrap.cc:492] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f: Bootstrap starting.
I20260620 14:46:17.725392 7731 tablet_bootstrap.cc:654] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:17.725648 7731 log.cc:824] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:17.725950 7732 tablet_bootstrap.cc:654] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:17.726266 7732 log.cc:824] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:17.726307 7731 tablet_bootstrap.cc:492] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3: No bootstrap required, opened a new log
I20260620 14:46:17.726359 7731 ts_tablet_manager.cc:1397] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:46:17.726850 7732 tablet_bootstrap.cc:492] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f: No bootstrap required, opened a new log
I20260620 14:46:17.726902 7732 ts_tablet_manager.cc:1397] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:46:17.727564 7734 tablet_bootstrap.cc:492] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d: Bootstrap starting.
I20260620 14:46:17.728471 7731 raft_consensus.cc:348] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7332d57250a4c4da5689a2dec888ff3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40747 } } peers { permanent_uuid: "7f50bab07914483c9672fb1b82ea9a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38171 } } peers { permanent_uuid: "e07ea2005fa84020a75becdaa5468e7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40311 } }
I20260620 14:46:17.728650 7734 tablet_bootstrap.cc:654] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:17.728647 7731 raft_consensus.cc:374] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:17.728684 7731 raft_consensus.cc:729] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7332d57250a4c4da5689a2dec888ff3, State: Initialized, Role: FOLLOWER
I20260620 14:46:17.728820 7731 consensus_queue.cc:260] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7332d57250a4c4da5689a2dec888ff3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40747 } } peers { permanent_uuid: "7f50bab07914483c9672fb1b82ea9a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38171 } } peers { permanent_uuid: "e07ea2005fa84020a75becdaa5468e7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40311 } }
I20260620 14:46:17.728904 7734 log.cc:824] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:17.729058 7731 ts_tablet_manager.cc:1428] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:46:17.729169 7705 heartbeater.cc:503] Master 127.0.98.62:40459 was elected leader, sending a full tablet report...
I20260620 14:46:17.729283 7732 raft_consensus.cc:348] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7332d57250a4c4da5689a2dec888ff3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40747 } } peers { permanent_uuid: "7f50bab07914483c9672fb1b82ea9a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38171 } } peers { permanent_uuid: "e07ea2005fa84020a75becdaa5468e7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40311 } }
I20260620 14:46:17.729480 7732 raft_consensus.cc:374] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:17.729521 7732 raft_consensus.cc:729] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f50bab07914483c9672fb1b82ea9a3f, State: Initialized, Role: FOLLOWER
I20260620 14:46:17.730085 7734 tablet_bootstrap.cc:492] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d: No bootstrap required, opened a new log
I20260620 14:46:17.730185 7734 ts_tablet_manager.cc:1397] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:46:17.729990 7732 consensus_queue.cc:260] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7332d57250a4c4da5689a2dec888ff3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40747 } } peers { permanent_uuid: "7f50bab07914483c9672fb1b82ea9a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38171 } } peers { permanent_uuid: "e07ea2005fa84020a75becdaa5468e7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40311 } }
I20260620 14:46:17.730526 7732 ts_tablet_manager.cc:1428] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:46:17.730551 7433 heartbeater.cc:503] Master 127.0.98.62:40459 was elected leader, sending a full tablet report...
I20260620 14:46:17.732115 7734 raft_consensus.cc:348] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7332d57250a4c4da5689a2dec888ff3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40747 } } peers { permanent_uuid: "7f50bab07914483c9672fb1b82ea9a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38171 } } peers { permanent_uuid: "e07ea2005fa84020a75becdaa5468e7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40311 } }
I20260620 14:46:17.732260 7734 raft_consensus.cc:374] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:17.732300 7734 raft_consensus.cc:729] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07ea2005fa84020a75becdaa5468e7d, State: Initialized, Role: FOLLOWER
I20260620 14:46:17.732386 7734 consensus_queue.cc:260] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7332d57250a4c4da5689a2dec888ff3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40747 } } peers { permanent_uuid: "7f50bab07914483c9672fb1b82ea9a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38171 } } peers { permanent_uuid: "e07ea2005fa84020a75becdaa5468e7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40311 } }
I20260620 14:46:17.732599 7569 heartbeater.cc:503] Master 127.0.98.62:40459 was elected leader, sending a full tablet report...
I20260620 14:46:17.732563 7734 ts_tablet_manager.cc:1428] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:46:17.736565 7739 raft_consensus.cc:482] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:46:17.736644 7739 raft_consensus.cc:504] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7332d57250a4c4da5689a2dec888ff3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40747 } } peers { permanent_uuid: "7f50bab07914483c9672fb1b82ea9a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38171 } } peers { permanent_uuid: "e07ea2005fa84020a75becdaa5468e7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40311 } }
I20260620 14:46:17.736909 7739 leader_election.cc:288] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7332d57250a4c4da5689a2dec888ff3 (127.0.98.3:40747), 7f50bab07914483c9672fb1b82ea9a3f (127.0.98.1:38171)
I20260620 14:46:17.740260 7572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.737055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40747 (local address 127.0.98.2:47373)
0620 14:46:17.737125 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.737133 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.737161 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.737363 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.737364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.737365 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:46:17.737426 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.737428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.738294 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.738295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.739587 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.739588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.740081 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.740082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.740136 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.740140 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.740178 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.740207 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":53}
I20260620 14:46:17.740319 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.737229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47373 (local address 127.0.98.3:40747)
0620 14:46:17.737272 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.737274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.737281 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:46:17.737301 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.737301 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.737309 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.737352 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:46:17.737491 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.738212 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.739668 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.740026 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.740077 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.740085 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.740200 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.740254 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:46:17.740608 7658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a841ddb651544210b74b5ab3985ebb80" candidate_uuid: "e07ea2005fa84020a75becdaa5468e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7332d57250a4c4da5689a2dec888ff3" is_pre_election: true
I20260620 14:46:17.740743 7658 raft_consensus.cc:2454] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07ea2005fa84020a75becdaa5468e7d in term 0.
I20260620 14:46:17.740931 7574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.737042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38171 (local address 127.0.98.2:59259)
0620 14:46:17.737092 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:17.737103 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:17.737144 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:17.737391 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:17.737391 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:17.737392 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:46:17.737433 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.737434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.738294 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.738295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.739574 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:17.739577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.740677 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.740678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:17.740763 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.740766 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:46:17.740819 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:46:17.740856 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":42}
I20260620 14:46:17.740978 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.737089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59259 (local address 127.0.98.1:38171)
0620 14:46:17.737145 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.737147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.737157 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:46:17.737200 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.737201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.737211 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.737264 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:46:17.737485 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.738226 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.740114 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.740609 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.740678 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.740688 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.740879 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.740921 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:46:17.740954 7455 leader_election.cc:302] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7332d57250a4c4da5689a2dec888ff3, e07ea2005fa84020a75becdaa5468e7d; no voters:
I20260620 14:46:17.741093 7739 raft_consensus.cc:2781] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:46:17.741153 7739 raft_consensus.cc:482] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:46:17.741173 7739 raft_consensus.cc:3037] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:17.741494 7386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a841ddb651544210b74b5ab3985ebb80" candidate_uuid: "e07ea2005fa84020a75becdaa5468e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f50bab07914483c9672fb1b82ea9a3f" is_pre_election: true
I20260620 14:46:17.741604 7386 raft_consensus.cc:2454] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07ea2005fa84020a75becdaa5468e7d in term 0.
I20260620 14:46:17.742079 7739 raft_consensus.cc:504] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7332d57250a4c4da5689a2dec888ff3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40747 } } peers { permanent_uuid: "7f50bab07914483c9672fb1b82ea9a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38171 } } peers { permanent_uuid: "e07ea2005fa84020a75becdaa5468e7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40311 } }
I20260620 14:46:17.742183 7739 leader_election.cc:288] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [CANDIDATE]: Term 1 election: Requested vote from peers a7332d57250a4c4da5689a2dec888ff3 (127.0.98.3:40747), 7f50bab07914483c9672fb1b82ea9a3f (127.0.98.1:38171)
I20260620 14:46:17.742302 7658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a841ddb651544210b74b5ab3985ebb80" candidate_uuid: "e07ea2005fa84020a75becdaa5468e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7332d57250a4c4da5689a2dec888ff3"
I20260620 14:46:17.742317 7386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a841ddb651544210b74b5ab3985ebb80" candidate_uuid: "e07ea2005fa84020a75becdaa5468e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f50bab07914483c9672fb1b82ea9a3f"
I20260620 14:46:17.742365 7658 raft_consensus.cc:3037] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:17.742381 7386 raft_consensus.cc:3037] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:17.742952 7658 raft_consensus.cc:2454] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07ea2005fa84020a75becdaa5468e7d in term 1.
I20260620 14:46:17.743023 7386 raft_consensus.cc:2454] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07ea2005fa84020a75becdaa5468e7d in term 1.
I20260620 14:46:17.743095 7455 leader_election.cc:302] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7332d57250a4c4da5689a2dec888ff3, e07ea2005fa84020a75becdaa5468e7d; no voters:
I20260620 14:46:17.743176 7739 raft_consensus.cc:2781] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:17.743309 7739 raft_consensus.cc:686] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [term 1 LEADER]: Becoming Leader. State: Replica: e07ea2005fa84020a75becdaa5468e7d, State: Running, Role: LEADER
I20260620 14:46:17.743378 7739 consensus_queue.cc:237] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7332d57250a4c4da5689a2dec888ff3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40747 } } peers { permanent_uuid: "7f50bab07914483c9672fb1b82ea9a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38171 } } peers { permanent_uuid: "e07ea2005fa84020a75becdaa5468e7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40311 } }
I20260620 14:46:17.744200 6468 catalog_manager.cc:5495] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d reported cstate change: term changed from 0 to 1, leader changed from <none> to e07ea2005fa84020a75becdaa5468e7d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e07ea2005fa84020a75becdaa5468e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7332d57250a4c4da5689a2dec888ff3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f50bab07914483c9672fb1b82ea9a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07ea2005fa84020a75becdaa5468e7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40311 } health_report { overall_health: HEALTHY } } }
I20260620 14:46:17.755575 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.753289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.0.98.2:40311)
0620 14:46:17.753333 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.753335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.753344 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:46:17.753426 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.753427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.753449 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.753518 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:46:17.753642 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.754100 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.754850 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.755053 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.755112 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.755259 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:46:17.755350 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:46:17.755446 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.755486 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.755507 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:46:17.757542 7658 raft_consensus.cc:1261] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3 [term 1 FOLLOWER]: Refusing update from remote peer e07ea2005fa84020a75becdaa5468e7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:17.757555 7386 raft_consensus.cc:1261] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f [term 1 FOLLOWER]: Refusing update from remote peer e07ea2005fa84020a75becdaa5468e7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:17.757862 7740 consensus_queue.cc:1035] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7332d57250a4c4da5689a2dec888ff3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:46:17.757979 7739 consensus_queue.cc:1035] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f50bab07914483c9672fb1b82ea9a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:46:17.760893 7745 mvcc.cc:205] Tried to move back new op lower bound from 7298935921692680192 to 7298935921637609472. Current Snapshot: MvccSnapshot[applied={T|T < 7298935921692680192}]
I20260620 14:46:17.761116 7744 mvcc.cc:205] Tried to move back new op lower bound from 7298935921692680192 to 7298935921637609472. Current Snapshot: MvccSnapshot[applied={T|T < 7298935921692680192}]
I20260620 14:46:17.762181 7743 mvcc.cc:205] Tried to move back new op lower bound from 7298935921692680192 to 7298935921637609472. Current Snapshot: MvccSnapshot[applied={T|T < 7298935921692680192}]
I20260620 14:46:17.764158 6466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:17.764246 6466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:17.765440 6466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d08956433a1142789be562fbf8eb6245: Sending DeleteTablet for 3 replicas of tablet a841ddb651544210b74b5ab3985ebb80
I20260620 14:46:17.765738 7366 tablet_service.cc:1514] Processing DeleteTablet for tablet a841ddb651544210b74b5ab3985ebb80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56598
I20260620 14:46:17.765761 7502 tablet_service.cc:1514] Processing DeleteTablet for tablet a841ddb651544210b74b5ab3985ebb80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57374
I20260620 14:46:17.765911 7638 tablet_service.cc:1514] Processing DeleteTablet for tablet a841ddb651544210b74b5ab3985ebb80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48372
I20260620 14:46:17.765977 7754 tablet_replica.cc:330] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d: stopping tablet replica
I20260620 14:46:17.766067 7754 raft_consensus.cc:2229] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:46:17.766178 7755 tablet_replica.cc:330] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3: stopping tablet replica
I20260620 14:46:17.766191 7754 raft_consensus.cc:2258] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:17.766325 7755 raft_consensus.cc:2229] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:46:17.766458 7755 raft_consensus.cc:2258] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:17.766672 7754 ts_tablet_manager.cc:1905] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:17.766934 7755 ts_tablet_manager.cc:1905] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:17.767409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 6436
I20260620 14:46:17.768925 7755 ts_tablet_manager.cc:1918] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:46:17.769060 7755 log.cc:1196] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/a841ddb651544210b74b5ab3985ebb80
I20260620 14:46:17.769204 7754 ts_tablet_manager.cc:1918] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:46:17.769282 7754 log.cc:1196] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/a841ddb651544210b74b5ab3985ebb80
I20260620 14:46:17.769485 7755 ts_tablet_manager.cc:1939] T a841ddb651544210b74b5ab3985ebb80 P a7332d57250a4c4da5689a2dec888ff3: Deleting consensus metadata
I20260620 14:46:17.769574 7754 ts_tablet_manager.cc:1939] T a841ddb651544210b74b5ab3985ebb80 P e07ea2005fa84020a75becdaa5468e7d: Deleting consensus metadata
W20260620 14:46:17.772701 7592 connection.cc:535] server connection from 127.0.0.1:48372 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 14:46:17.772908 7456 connection.cc:535] server connection from 127.0.0.1:57374 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:46:17.775530 7753 tablet_replica.cc:330] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f: stopping tablet replica
I20260620 14:46:17.775661 7753 raft_consensus.cc:2229] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:46:17.775770 7753 raft_consensus.cc:2258] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:17.780161 7753 ts_tablet_manager.cc:1905] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:17.781477 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.774020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44942 (local address 127.0.98.61:35981)
0620 14:46:17.774058 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.774060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.774338 (+ 278us) server_negotiation.cc:407] Connection header received
0620 14:46:17.774399 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.774400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.774411 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.774462 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.774972 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.775724 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.776756 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.777047 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.777117 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.778773 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.778774 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.778775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.778782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.780361 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.780721 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.780723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.780723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.780784 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.780970 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.780971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.780972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.781100 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.781198 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.781298 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.781373 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:46:17.781847 7753 ts_tablet_manager.cc:1918] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:46:17.781920 7753 log.cc:1196] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0/minicluster-data/ts-0/wal/wals/a841ddb651544210b74b5ab3985ebb80
I20260620 14:46:17.782179 7753 ts_tablet_manager.cc:1939] T a841ddb651544210b74b5ab3985ebb80 P 7f50bab07914483c9672fb1b82ea9a3f: Deleting consensus metadata
W20260620 14:46:17.782552 7320 connection.cc:441] server connection from 127.0.0.1:56598 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56598 (request call id 1) could send its response
I20260620 14:46:17.782770 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:17.774189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51276 (local address 127.0.98.60:35679)
0620 14:46:17.774226 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:17.774227 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:17.775601 (+ 1374us) server_negotiation.cc:407] Connection header received
0620 14:46:17.775714 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:17.775715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:17.775725 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:17.775773 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:17.776357 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.776822 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.777614 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:17.777812 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:17.777863 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:17.778963 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:17.778965 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:17.778965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:17.778971 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:17.780403 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.782007 (+ 1604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.782008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.782008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.782081 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:17.782276 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:17.782277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:17.782278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:17.782380 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:17.782432 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:17.782603 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 14:46:17.782679 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:46:18.763355 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:18.762706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:34637)
0620 14:46:18.763211 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:18.763296 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":55,"threads_started":1}
I20260620 14:46:18.763396 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:18.762840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:53237)
0620 14:46:18.763277 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:18.763356 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":52,"threads_started":1}
W20260620 14:46:18.763657 7433 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40459 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
I20260620 14:46:18.763680 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:18.762489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:45875)
0620 14:46:18.763575 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:18.763640 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1069,"thread_start_us":260,"threads_started":1}
W20260620 14:46:18.763654 7705 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40459 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
W20260620 14:46:18.763830 7569 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40459 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
I20260620 14:46:19.266433 7766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d08956433a1142789be562fbf8eb6245)
I20260620 14:46:19.266556 7766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: false } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } }
I20260620 14:46:19.266749 7766 leader_election.cc:288] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d08956433a1142789be562fbf8eb6245 (127.0.98.62:40459)
I20260620 14:46:19.267066 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:19.266880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.0.1:38198)
0620 14:46:19.266982 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:19.267029 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":55,"threads_started":1}
W20260620 14:46:19.267232 7053 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
I20260620 14:46:19.267613 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:19.267521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.0.1:38208)
0620 14:46:19.267561 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:19.267591 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 14:46:19.267702 7053 leader_election.cc:334] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d08956433a1142789be562fbf8eb6245 (127.0.98.62:40459): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
I20260620 14:46:19.267763 7053 leader_election.cc:302] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6604b93b1d5b4df4963aea5f8e50a62a; no voters: d08956433a1142789be562fbf8eb6245
I20260620 14:46:19.267877 7766 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:46:19.764124 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:19.763931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:53541)
0620 14:46:19.764043 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:19.764079 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
I20260620 14:46:19.764254 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:19.763931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:49219)
0620 14:46:19.764185 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:19.764212 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":65,"threads_started":1}
I20260620 14:46:19.764254 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:19.764078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:44685)
0620 14:46:19.764179 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:19.764212 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":38,"threads_started":1}
I20260620 14:46:20.764845 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:20.764643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:41219)
0620 14:46:20.764762 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:20.764794 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260620 14:46:20.764895 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:20.764665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:38169)
0620 14:46:20.764803 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:20.764835 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260620 14:46:20.764881 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:20.764643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:54619)
0620 14:46:20.764803 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:20.764835 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":65,"threads_started":1}
I20260620 14:46:21.435849 7775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d08956433a1142789be562fbf8eb6245)
I20260620 14:46:21.435945 7775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: false } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } }
I20260620 14:46:21.436064 7775 leader_election.cc:288] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d08956433a1142789be562fbf8eb6245 (127.0.98.62:40459)
I20260620 14:46:21.436395 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:21.436222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.0.1:38466)
0620 14:46:21.436315 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:21.436358 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":44,"threads_started":1}
I20260620 14:46:21.436749 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:21.436659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.0.1:38470)
0620 14:46:21.436707 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:21.436730 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 14:46:21.436818 7053 leader_election.cc:334] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d08956433a1142789be562fbf8eb6245 (127.0.98.62:40459): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
I20260620 14:46:21.436864 7053 leader_election.cc:302] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6604b93b1d5b4df4963aea5f8e50a62a; no voters: d08956433a1142789be562fbf8eb6245
I20260620 14:46:21.436918 7775 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:46:21.765467 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:21.765244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:43197)
0620 14:46:21.765409 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:21.765433 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":56,"threads_started":1}
I20260620 14:46:21.765445 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:21.765225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:60895)
0620 14:46:21.765353 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:21.765390 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260620 14:46:21.765607 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:21.765423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:47751)
0620 14:46:21.765527 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:21.765560 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260620 14:46:22.769596 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:22.769313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:48083)
0620 14:46:22.769504 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:22.769541 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":54,"threads_started":1}
I20260620 14:46:22.769881 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:22.769398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:49391)
0620 14:46:22.769794 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:22.769841 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":42,"threads_started":1}
I20260620 14:46:22.770246 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:22.769107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:54409)
0620 14:46:22.770147 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:22.770201 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1018,"thread_start_us":86,"threads_started":1}
I20260620 14:46:23.484867 7784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d08956433a1142789be562fbf8eb6245)
I20260620 14:46:23.484958 7784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: false } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } }
I20260620 14:46:23.485097 7784 leader_election.cc:288] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d08956433a1142789be562fbf8eb6245 (127.0.98.62:40459)
I20260620 14:46:23.485404 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:23.485228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.0.1:38668)
0620 14:46:23.485323 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:23.485366 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":43,"threads_started":1}
I20260620 14:46:23.485718 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:23.485644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.0.1:38674)
0620 14:46:23.485678 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:23.485700 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 14:46:23.485774 7053 leader_election.cc:334] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d08956433a1142789be562fbf8eb6245 (127.0.98.62:40459): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
I20260620 14:46:23.485826 7053 leader_election.cc:302] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6604b93b1d5b4df4963aea5f8e50a62a; no voters: d08956433a1142789be562fbf8eb6245
I20260620 14:46:23.485894 7784 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:46:23.770772 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:23.770329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:44793)
0620 14:46:23.770674 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:23.770704 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":91,"threads_started":1}
I20260620 14:46:23.770803 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:23.770608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:48213)
0620 14:46:23.770719 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:23.770756 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260620 14:46:23.770803 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:23.770578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:58245)
0620 14:46:23.770719 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:23.770756 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":49,"threads_started":1}
I20260620 14:46:24.771397 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:24.771192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:40609)
0620 14:46:24.771306 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:24.771344 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20260620 14:46:24.771564 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:24.771165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:34805)
0620 14:46:24.771490 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:24.771524 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":93,"threads_started":1}
I20260620 14:46:24.771737 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:24.771165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:35529)
0620 14:46:24.771669 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:24.771701 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":72,"threads_started":1}
I20260620 14:46:25.772007 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:25.771776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:47383)
0620 14:46:25.771910 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:25.771950 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260620 14:46:25.772069 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:25.771906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:44787)
0620 14:46:25.771998 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:25.772030 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":54,"threads_started":1}
I20260620 14:46:25.772383 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:25.772018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:57941)
0620 14:46:25.772311 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:25.772344 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":56,"threads_started":1}
I20260620 14:46:26.772573 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:26.772343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:34171)
0620 14:46:26.772472 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:26.772517 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260620 14:46:26.772706 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:26.772513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:46325)
0620 14:46:26.772632 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:26.772661 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260620 14:46:26.772826 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:26.772673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:47127)
0620 14:46:26.772763 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:26.772790 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":41,"threads_started":1}
I20260620 14:46:26.780756 7799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d08956433a1142789be562fbf8eb6245)
I20260620 14:46:26.780836 7799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: false } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } }
I20260620 14:46:26.780942 7799 leader_election.cc:288] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d08956433a1142789be562fbf8eb6245 (127.0.98.62:40459)
I20260620 14:46:26.781214 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:26.781051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.0.1:46858)
0620 14:46:26.781151 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:26.781184 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":62,"threads_started":1}
W20260620 14:46:26.781280 7053 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260620 14:46:26.781558 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:26.781502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.0.1:46866)
0620 14:46:26.781523 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:26.781544 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260620 14:46:26.781615 7053 leader_election.cc:334] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d08956433a1142789be562fbf8eb6245 (127.0.98.62:40459): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
I20260620 14:46:26.781659 7053 leader_election.cc:302] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6604b93b1d5b4df4963aea5f8e50a62a; no voters: d08956433a1142789be562fbf8eb6245
I20260620 14:46:26.781725 7799 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:46:27.773150 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:27.772932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:35207)
0620 14:46:27.773057 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:27.773099 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260620 14:46:27.773305 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:27.773049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:36555)
0620 14:46:27.773182 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:27.773262 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260620 14:46:27.773305 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:27.773134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:54727)
0620 14:46:27.773237 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:27.773263 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":42,"threads_started":1}
I20260620 14:46:28.773797 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:28.773562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:48323)
0620 14:46:28.773688 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:28.773732 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260620 14:46:28.773801 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:28.773649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:52873)
0620 14:46:28.773729 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:28.773761 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65,"thread_start_us":41,"threads_started":1}
I20260620 14:46:28.773965 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:28.773744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:36019)
0620 14:46:28.773850 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:28.773927 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260620 14:46:29.774391 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:29.774182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:49281)
0620 14:46:29.774299 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:29.774338 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
I20260620 14:46:29.774391 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:29.774195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:58527)
0620 14:46:29.774323 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:29.774353 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260620 14:46:29.774569 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:29.774329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:39719)
0620 14:46:29.774418 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:29.774516 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":39,"threads_started":1}
I20260620 14:46:30.774969 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:30.774766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:60071)
0620 14:46:30.774881 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:30.774919 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":62,"threads_started":1}
I20260620 14:46:30.774969 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:30.774766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:48305)
0620 14:46:30.774895 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:30.774924 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260620 14:46:30.775072 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:30.774898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:47631)
0620 14:46:30.774999 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:30.775028 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":42,"threads_started":1}
I20260620 14:46:30.790091 7814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d08956433a1142789be562fbf8eb6245)
I20260620 14:46:30.790179 7814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: false } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } }
I20260620 14:46:30.790293 7814 leader_election.cc:288] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d08956433a1142789be562fbf8eb6245 (127.0.98.62:40459)
I20260620 14:46:30.790575 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:30.790422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.0.1:47090)
0620 14:46:30.790506 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:30.790543 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":41,"threads_started":1}
I20260620 14:46:30.790915 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:30.790849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.0.1:47096)
0620 14:46:30.790883 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:30.790903 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 14:46:30.790982 7053 leader_election.cc:334] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d08956433a1142789be562fbf8eb6245 (127.0.98.62:40459): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
I20260620 14:46:30.791041 7053 leader_election.cc:302] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6604b93b1d5b4df4963aea5f8e50a62a; no voters: d08956433a1142789be562fbf8eb6245
I20260620 14:46:30.791101 7814 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:46:31.775691 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:31.775432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:53145)
0620 14:46:31.775599 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:31.775640 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":99,"threads_started":1}
I20260620 14:46:31.775691 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:31.775445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:54101)
0620 14:46:31.775591 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:31.775637 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260620 14:46:31.775905 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:31.775450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:39609)
0620 14:46:31.775829 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:31.775863 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":76,"threads_started":1}
I20260620 14:46:32.776351 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:32.776151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:43243)
0620 14:46:32.776276 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:32.776311 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260620 14:46:32.776372 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:32.776152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:47433)
0620 14:46:32.776289 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:32.776327 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":56,"threads_started":1}
I20260620 14:46:32.776412 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:32.776263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:48181)
0620 14:46:32.776353 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:32.776377 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":43,"threads_started":1}
I20260620 14:46:33.776918 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:33.776708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:40349)
0620 14:46:33.776847 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:33.776878 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":52,"threads_started":1}
I20260620 14:46:33.776918 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:33.776713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:57843)
0620 14:46:33.776821 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:33.776863 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":59,"threads_started":1}
I20260620 14:46:33.777012 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:33.776851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:42445)
0620 14:46:33.776937 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:33.776969 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":38,"threads_started":1}
I20260620 14:46:34.777524 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:34.777335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:36797)
0620 14:46:34.777444 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:34.777481 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
I20260620 14:46:34.777594 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:34.777426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:45857)
0620 14:46:34.777520 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:34.777552 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":43,"threads_started":1}
I20260620 14:46:34.777714 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:34.777323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:54265)
0620 14:46:34.777644 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:34.777677 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":70,"threads_started":1}
I20260620 14:46:35.778143 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:35.777886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:49249)
0620 14:46:35.778028 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:35.778074 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260620 14:46:35.778239 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:35.778056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:50499)
0620 14:46:35.778170 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:35.778194 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":44,"threads_started":1}
I20260620 14:46:35.778306 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:35.778036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:47219)
0620 14:46:35.778156 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:35.778194 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260620 14:46:36.778784 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:36.778558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:33957)
0620 14:46:36.778682 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:36.778733 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260620 14:46:36.778795 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:36.778553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:44459)
0620 14:46:36.778707 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:36.778744 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20260620 14:46:36.778810 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:36.778636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:37983)
0620 14:46:36.778744 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:36.778776 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260620 14:46:37.779347 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:37.779151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:46647)
0620 14:46:37.779269 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:37.779303 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260620 14:46:37.779350 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:37.779151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:45027)
0620 14:46:37.779277 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:37.779307 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260620 14:46:37.779350 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:37.779147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:42129)
0620 14:46:37.779270 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:37.779306 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260620 14:46:38.779937 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:38.779717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:33597)
0620 14:46:38.779853 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:38.779893 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260620 14:46:38.779953 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:38.779790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:39547)
0620 14:46:38.779887 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:38.779917 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":57,"threads_started":1}
I20260620 14:46:38.779937 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:38.779733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:56553)
0620 14:46:38.779846 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:38.779883 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260620 14:46:39.780555 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:39.780355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:38415)
0620 14:46:39.780472 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:39.780506 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":52,"threads_started":1}
I20260620 14:46:39.780555 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:39.780331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:58689)
0620 14:46:39.780472 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:39.780506 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260620 14:46:39.780558 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:39.780355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:54877)
0620 14:46:39.780476 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:39.780511 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":52,"threads_started":1}
I20260620 14:46:40.643954 7844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d08956433a1142789be562fbf8eb6245)
I20260620 14:46:40.644054 7844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d08956433a1142789be562fbf8eb6245" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } peers { permanent_uuid: "6604b93b1d5b4df4963aea5f8e50a62a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35981 } attrs { promote: false } } peers { permanent_uuid: "cc53f0865d504188a6310363ab89733b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35679 } attrs { promote: true } }
I20260620 14:46:40.644172 7844 leader_election.cc:288] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d08956433a1142789be562fbf8eb6245 (127.0.98.62:40459)
I20260620 14:46:40.644490 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:40.644317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.0.1:48060)
0620 14:46:40.644417 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:40.644452 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":50,"threads_started":1}
W20260620 14:46:40.644564 7053 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260620 14:46:40.644881 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:40.644800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.0.1:48064)
0620 14:46:40.644840 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:40.644864 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 14:46:40.644956 7053 leader_election.cc:334] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d08956433a1142789be562fbf8eb6245 (127.0.98.62:40459): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
I20260620 14:46:40.645004 7053 leader_election.cc:302] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6604b93b1d5b4df4963aea5f8e50a62a; no voters: d08956433a1142789be562fbf8eb6245
I20260620 14:46:40.645057 7844 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6604b93b1d5b4df4963aea5f8e50a62a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:46:40.781131 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:40.780929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:43003)
0620 14:46:40.781049 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:40.781081 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260620 14:46:40.781131 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:40.780929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:43913)
0620 14:46:40.781049 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:40.781081 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
I20260620 14:46:40.781133 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:40.780934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:34619)
0620 14:46:40.781049 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:40.781081 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260620 14:46:41.781656 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:41.781479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:51401)
0620 14:46:41.781591 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:41.781621 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":51,"threads_started":1}
I20260620 14:46:41.781656 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:41.781457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:52727)
0620 14:46:41.781562 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:41.781601 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":60,"threads_started":1}
I20260620 14:46:41.781720 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:41.781557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:33763)
0620 14:46:41.781647 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:41.781680 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":40,"threads_started":1}
I20260620 14:46:42.782286 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:42.782087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:34631)
0620 14:46:42.782195 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:42.782233 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260620 14:46:42.782286 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:42.782092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:54037)
0620 14:46:42.782216 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:42.782245 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":55,"threads_started":1}
I20260620 14:46:42.782496 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:42.782077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:56915)
0620 14:46:42.782416 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:42.782459 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":66,"threads_started":1}
I20260620 14:46:43.782914 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:43.782705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:52423)
0620 14:46:43.782824 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:43.782865 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260620 14:46:43.782914 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:43.782692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:55565)
0620 14:46:43.782834 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:43.782865 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260620 14:46:43.783054 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:43.782826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:57741)
0620 14:46:43.782941 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:43.782977 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260620 14:46:44.783535 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:44.783337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:38763)
0620 14:46:44.783458 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:44.783493 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":43,"threads_started":1}
I20260620 14:46:44.783562 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:44.783313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:51591)
0620 14:46:44.783460 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:44.783500 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
I20260620 14:46:44.783619 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:44.783441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:34055)
0620 14:46:44.783555 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:44.783583 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260620 14:46:45.784099 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:45.783901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:47369)
0620 14:46:45.784015 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:45.784055 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
I20260620 14:46:45.784153 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:45.783959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:44585)
0620 14:46:45.784074 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:45.784110 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260620 14:46:45.784153 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:45.783959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:59677)
0620 14:46:45.784086 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:45.784114 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":52,"threads_started":1}
I20260620 14:46:46.784700 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:46.784490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:57091)
0620 14:46:46.784605 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:46.784647 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
I20260620 14:46:46.784701 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:46.784486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:46035)
0620 14:46:46.784618 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:46.784657 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260620 14:46:46.784708 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:46.784505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:44329)
0620 14:46:46.784629 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:46.784656 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":49,"threads_started":1}
I20260620 14:46:47.785317 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:47.785130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:53909)
0620 14:46:47.785240 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:47.785274 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
I20260620 14:46:47.785329 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:47.785138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:46203)
0620 14:46:47.785254 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:47.785286 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":51,"threads_started":1}
I20260620 14:46:47.785372 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:47.785145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:55867)
0620 14:46:47.785253 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:47.785286 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":58,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.98.62:40459,127.0.98.61:35981,127.0.98.60:35679, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.98.62:40459,127.0.98.61:35981,127.0.98.60:35679, num_attempts: 1)
I20260620 14:46:47.938278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 7303
I20260620 14:46:47.942857 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 7439
I20260620 14:46:47.947470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 7575
I20260620 14:46:47.951198 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 6932
I20260620 14:46:47.954871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 7120
2026-06-20T14:46:47Z chronyd exiting
I20260620 14:46:47.969570 392 test_util.cc:175] -----------------------------------------------
I20260620 14:46:47.969631 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966748446763-392-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (32884 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:46:48 dist-test-slave-cx89 krb5kdc[7873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:46:48 dist-test-slave-cx89 krb5kdc[7873](info): set up 2 sockets
Jun 20 14:46:48 dist-test-slave-cx89 krb5kdc[7873](info): commencing operation
krb5kdc: starting...
W20260620 14:46:48.907521 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:46:48 dist-test-slave-cx89 krb5kdc[7873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:46:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:46:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 14:46:49.017691 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41175
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:49.095772 7886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:49.095980 7886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:49.096009 7886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:49.097123 7886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:46:49.097160 7886 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:46:49.097177 7886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:49.097194 7886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:49.097210 7886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:46:49.097231 7886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:46:49.098623 7886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41175
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:49.098850 7886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:49.099041 7886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:49.103466 7894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:49.103490 7892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:49.103559 7891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:49.103569 7886 server_base.cc:1034] running on GCE node
I20260620 14:46:49.103907 7886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:49.104172 7886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:49.105304 7886 hybrid_clock.cc:648] HybridClock initialized: now 1781966809105285 us; error 33 us; skew 500 ppm
Jun 20 14:46:49 dist-test-slave-cx89 krb5kdc[7873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966809, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:49.107627 7886 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 14:46:49.107997 7886 webserver.cc:460] Webserver started at http://127.0.98.62:45513/ using document root <none> and password file <none>
I20260620 14:46:49.108181 7886 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:49.108218 7886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:49.108304 7886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:49.109248 7886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "593202c7340441279dcc0d96ca78f299"
format_stamp: "Formatted at 2026-06-20 14:46:49 on dist-test-slave-cx89"
server_key: "c0465da7091adc2542d4d6e590ebd3de"
server_key_iv: "68bfee979f915904f53ed2ee3d960077"
server_key_version: "encryptionkey@0"
I20260620 14:46:49.109520 7886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "593202c7340441279dcc0d96ca78f299"
format_stamp: "Formatted at 2026-06-20 14:46:49 on dist-test-slave-cx89"
server_key: "c0465da7091adc2542d4d6e590ebd3de"
server_key_iv: "68bfee979f915904f53ed2ee3d960077"
server_key_version: "encryptionkey@0"
I20260620 14:46:49.110636 7886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:46:49.111254 7901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:49.111397 7886 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:49.111495 7886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "593202c7340441279dcc0d96ca78f299"
format_stamp: "Formatted at 2026-06-20 14:46:49 on dist-test-slave-cx89"
server_key: "c0465da7091adc2542d4d6e590ebd3de"
server_key_iv: "68bfee979f915904f53ed2ee3d960077"
server_key_version: "encryptionkey@0"
I20260620 14:46:49.111546 7886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:49.122380 7886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:49.124382 7886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:49.124490 7886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:49.128631 7886 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41175
I20260620 14:46:49.128697 7963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41175 every 8 connection(s)
I20260620 14:46:49.128990 7886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:46:49.129644 7964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:49.131819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 7886
I20260620 14:46:49.131912 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:46:49.131958 7964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299: Bootstrap starting.
I20260620 14:46:49.132097 392 external_mini_cluster.cc:1428] Setting key ea6c778d2330f60f68fefccfbac1f9f4
I20260620 14:46:49.132582 7964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:49.132894 7964 log.cc:824] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:49.133502 7964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299: No bootstrap required, opened a new log
I20260620 14:46:49.135645 7964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593202c7340441279dcc0d96ca78f299" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41175 } }
I20260620 14:46:49.135823 7964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:49.135856 7964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593202c7340441279dcc0d96ca78f299, State: Initialized, Role: FOLLOWER
I20260620 14:46:49.135977 7964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593202c7340441279dcc0d96ca78f299" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41175 } }
I20260620 14:46:49.136045 7964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:46:49.136118 7964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:46:49.136166 7964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:49.136675 7964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593202c7340441279dcc0d96ca78f299" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41175 } }
I20260620 14:46:49.136783 7964 leader_election.cc:302] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 593202c7340441279dcc0d96ca78f299; no voters:
I20260620 14:46:49.136930 7964 leader_election.cc:288] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:46:49.136965 7969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:49.137115 7969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299 [term 1 LEADER]: Becoming Leader. State: Replica: 593202c7340441279dcc0d96ca78f299, State: Running, Role: LEADER
I20260620 14:46:49.137189 7964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:46:49.137213 7969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593202c7340441279dcc0d96ca78f299" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41175 } }
I20260620 14:46:49.137583 7969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "593202c7340441279dcc0d96ca78f299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593202c7340441279dcc0d96ca78f299" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41175 } } }
I20260620 14:46:49.137871 7969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299 [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:49.138008 7969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 593202c7340441279dcc0d96ca78f299. Latest consensus state: current_term: 1 leader_uuid: "593202c7340441279dcc0d96ca78f299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593202c7340441279dcc0d96ca78f299" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41175 } } }
I20260620 14:46:49.138069 7969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299 [sys.catalog]: This master's current role is: LEADER
Jun 20 14:46:49 dist-test-slave-cx89 krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
W20260620 14:46:49.138866 7985 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:46:49.138916 7985 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:46:49.138963 7978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:46:49.139143 7978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:46:49.143213 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.132614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60248 (local address 127.0.98.62:41175)
0620 14:46:49.132770 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:49.132778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:49.132882 (+ 104us) server_negotiation.cc:407] Connection header received
0620 14:46:49.133357 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:49.133363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:49.133410 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:49.133523 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:49.134228 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.135080 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.136112 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.136556 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.136637 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.139228 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:49.139232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:49.139232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:49.139243 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:49.141217 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:49.141599 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:49.141602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:49.141602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:49.141703 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:49.141895 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:49.141896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:49.141896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:49.142216 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:49.142364 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:49.142851 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 14:46:49.142980 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260620 14:46:49.144164 7978 catalog_manager.cc:1347] Generated new cluster ID: d477e9ee3180401d936b9beadf7203a2
I20260620 14:46:49.144218 7978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:46:49.156935 7978 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:46:49.157653 7978 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:46:49.168395 7978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299: Generated new TSK 0
I20260620 14:46:49.168583 7978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 14:46:49.199985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41175
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:49.278386 7990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:49.278586 7990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:49.278609 7990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:49.279769 7990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:49.279810 7990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:49.279829 7990 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:46:49.279844 7990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:46:49.281288 7990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:49.281508 7990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:49.281730 7990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:49.286393 7998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:49.286420 7995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:49.286495 7990 server_base.cc:1034] running on GCE node
W20260620 14:46:49.286401 7996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:49.286844 7990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:49.287062 7990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:49.288203 7990 hybrid_clock.cc:648] HybridClock initialized: now 1781966809288193 us; error 25 us; skew 500 ppm
Jun 20 14:46:49 dist-test-slave-cx89 krb5kdc[7873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966809, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:49.290532 7990 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:46:49.290901 7990 webserver.cc:460] Webserver started at http://127.0.98.1:41595/ using document root <none> and password file <none>
I20260620 14:46:49.291079 7990 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:49.291121 7990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:49.291208 7990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:49.292230 7990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "1978987151464256ba8d257d1ff48667"
format_stamp: "Formatted at 2026-06-20 14:46:49 on dist-test-slave-cx89"
server_key: "9e99715bd62f1c6c4920f85406618479"
server_key_iv: "2fcc8f0ed467d046b73acbd4891e8173"
server_key_version: "encryptionkey@0"
I20260620 14:46:49.292507 7990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1978987151464256ba8d257d1ff48667"
format_stamp: "Formatted at 2026-06-20 14:46:49 on dist-test-slave-cx89"
server_key: "9e99715bd62f1c6c4920f85406618479"
server_key_iv: "2fcc8f0ed467d046b73acbd4891e8173"
server_key_version: "encryptionkey@0"
I20260620 14:46:49.293645 7990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:46:49.294265 8005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:49.294402 7990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:46:49.294479 7990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "1978987151464256ba8d257d1ff48667"
format_stamp: "Formatted at 2026-06-20 14:46:49 on dist-test-slave-cx89"
server_key: "9e99715bd62f1c6c4920f85406618479"
server_key_iv: "2fcc8f0ed467d046b73acbd4891e8173"
server_key_version: "encryptionkey@0"
I20260620 14:46:49.294533 7990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:49.321018 7990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:49.322927 7990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:49.323032 7990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:49.323526 7990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:49.323572 7990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:49.323593 7990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:49.323608 7990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:49.330195 7990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43177
I20260620 14:46:49.330307 8120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43177 every 8 connection(s)
Jun 20 14:46:49 dist-test-slave-cx89 krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966809, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:46:49.330623 7990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:46:49.333402 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.323924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39607 (local address 127.0.98.62:41175)
0620 14:46:49.323985 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:49.323987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:49.324236 (+ 249us) server_negotiation.cc:407] Connection header received
0620 14:46:49.324889 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:49.324891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:49.324903 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:49.324959 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:49.325798 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.326648 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.328185 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.328490 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.328546 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.331038 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:49.331039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:49.331040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:49.331058 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:49.332297 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:49.332633 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:49.332634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:49.332634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:49.332680 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:49.332920 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:49.332921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:49.332922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:49.333017 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:49.333074 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:49.333278 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 14:46:49.333339 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:46:49.333570 8014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.323908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41175 (local address 127.0.98.1:39607)
0620 14:46:49.324102 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:49.324148 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:49.324777 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:49.325044 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:49.325049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:49.325406 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:49.325693 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.325699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.326745 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.326746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.328093 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.328096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.328570 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.328572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.328721 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.329268 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:49.329278 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:49.330927 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:49.332367 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:49.332371 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:49.332373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:49.332579 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:49.332754 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:49.332755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:49.332756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:49.332865 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:49.333059 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:49.333061 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:49.333186 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 14:46:49.333259 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:46:49.333383 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260620 14:46:49.334025 8121 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41175
I20260620 14:46:49.334110 8121 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:49.334276 8121 heartbeater.cc:511] Master 127.0.98.62:41175 requested a full tablet report, sending...
I20260620 14:46:49.334336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 7990
I20260620 14:46:49.334440 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:46:49.334654 392 external_mini_cluster.cc:1428] Setting key b4b35b71fc053646630ad27e2c4bae53
I20260620 14:46:49.334766 7917 ts_manager.cc:194] Registered new tserver with Master: 1978987151464256ba8d257d1ff48667 (127.0.98.1:43177)
I20260620 14:46:49.335603 7917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39607
I20260620 14:46:49.340304 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.335196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34605 (local address 127.0.98.62:41175)
0620 14:46:49.335228 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:49.335230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:49.335237 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:46:49.335256 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:49.335257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:49.335266 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:49.335313 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:49.335657 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.336308 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.336966 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.337171 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.337222 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.338111 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:49.338112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:49.338112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:49.338118 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:49.339311 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:49.339634 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:49.339635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:49.339635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:49.339688 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:49.339842 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:49.339843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:49.339843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:49.339945 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:49.339998 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:49.340142 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 14:46:49.340214 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:46:49.340317 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.335015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41175 (local address 127.0.98.1:34605)
0620 14:46:49.335159 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:49.335170 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:49.335218 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:49.335317 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:49.335317 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:49.335506 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:49.335584 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.335586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.336376 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.336377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.336896 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.336898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.337227 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.337228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.337289 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.337727 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:49.337732 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:49.338039 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:49.339381 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:49.339382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:49.339383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:49.339568 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:49.339738 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:49.339739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:49.339739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:49.339795 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:49.339995 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:49.339996 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:49.340038 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:46:49.340114 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:46:49.340217 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 14:46:49.358345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41175
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:49.436897 8130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:49.437165 8130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:49.437198 8130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:49.438392 8130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:49.438434 8130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:49.438454 8130 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:46:49.438472 8130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:46:49.439980 8130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:49.440196 8130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:49.440402 8130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:49.445258 8138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:49.445359 8130 server_base.cc:1034] running on GCE node
W20260620 14:46:49.445225 8136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:49.445278 8135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:49.445633 8130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:49.445858 8130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:49.446992 8130 hybrid_clock.cc:648] HybridClock initialized: now 1781966809446982 us; error 32 us; skew 500 ppm
Jun 20 14:46:49 dist-test-slave-cx89 krb5kdc[7873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966809, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:49.449222 8130 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:46:49.449620 8130 webserver.cc:460] Webserver started at http://127.0.98.2:34683/ using document root <none> and password file <none>
I20260620 14:46:49.449815 8130 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:49.449860 8130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:49.449980 8130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:49.450999 8130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "bab203b9622e488c9001e049938329c4"
format_stamp: "Formatted at 2026-06-20 14:46:49 on dist-test-slave-cx89"
server_key: "54b4ae19a670daabc33305927dddb332"
server_key_iv: "8c285cc4a396dda9f2bf3940989b98f0"
server_key_version: "encryptionkey@0"
I20260620 14:46:49.451320 8130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bab203b9622e488c9001e049938329c4"
format_stamp: "Formatted at 2026-06-20 14:46:49 on dist-test-slave-cx89"
server_key: "54b4ae19a670daabc33305927dddb332"
server_key_iv: "8c285cc4a396dda9f2bf3940989b98f0"
server_key_version: "encryptionkey@0"
I20260620 14:46:49.452569 8130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:46:49.453187 8145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:49.453332 8130 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:46:49.453410 8130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "bab203b9622e488c9001e049938329c4"
format_stamp: "Formatted at 2026-06-20 14:46:49 on dist-test-slave-cx89"
server_key: "54b4ae19a670daabc33305927dddb332"
server_key_iv: "8c285cc4a396dda9f2bf3940989b98f0"
server_key_version: "encryptionkey@0"
I20260620 14:46:49.453465 8130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:49.463769 8130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:49.465451 8130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:49.465575 8130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:49.466038 8130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:49.466079 8130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:49.466107 8130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:49.466132 8130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:46:49 dist-test-slave-cx89 krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966809, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:46:49.472540 8130 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45177
I20260620 14:46:49.472620 8260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45177 every 8 connection(s)
I20260620 14:46:49.472929 8130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:46:49.475406 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.466474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33813 (local address 127.0.98.62:41175)
0620 14:46:49.466523 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:49.466525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:49.466715 (+ 190us) server_negotiation.cc:407] Connection header received
0620 14:46:49.467212 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:49.467214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:49.467227 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:49.467293 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:49.468174 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.468943 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.470024 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.470324 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.470386 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.472851 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:49.472852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:49.472852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:49.472859 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:49.474195 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:49.474547 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:49.474549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:49.474549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:49.474605 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:49.474813 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:49.474814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:49.474814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:49.474929 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:49.474984 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:49.475241 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 14:46:49.475317 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:46:49.475600 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.466473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41175 (local address 127.0.98.2:33813)
0620 14:46:49.466605 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:49.466638 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:49.467117 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:49.467405 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:49.467410 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:49.467769 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:49.468061 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.468068 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.469023 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.469024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.469938 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.469941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.470415 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.470416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.470572 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.471049 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:49.471060 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:49.472762 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:49.474269 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:49.474275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:49.474277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:49.474482 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:49.474655 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:49.474656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:49.474656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:49.474758 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:49.474981 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:49.474986 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:49.475117 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 14:46:49.475209 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 14:46:49.475362 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":72,"threads_started":1}
I20260620 14:46:49.476027 8261 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41175
I20260620 14:46:49.476101 8261 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:49.476274 8261 heartbeater.cc:511] Master 127.0.98.62:41175 requested a full tablet report, sending...
I20260620 14:46:49.476641 7918 ts_manager.cc:194] Registered new tserver with Master: bab203b9622e488c9001e049938329c4 (127.0.98.2:45177)
I20260620 14:46:49.477099 7918 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33813
I20260620 14:46:49.481990 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.477085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34389 (local address 127.0.98.62:41175)
0620 14:46:49.477132 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:49.477134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:49.477211 (+ 77us) server_negotiation.cc:407] Connection header received
0620 14:46:49.477284 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:49.477285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:49.477294 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:49.477333 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:49.477623 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.478091 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.478746 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.478940 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.478989 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.479799 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:49.479801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:49.479801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:49.479807 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:49.480997 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:49.481301 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:49.481302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:49.481302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:49.481355 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:49.481506 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:49.481507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:49.481508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:49.481641 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:49.481705 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:49.481836 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 14:46:49.481906 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:46:49.482004 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.476998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41175 (local address 127.0.98.2:34389)
0620 14:46:49.477151 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:49.477161 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:49.477237 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:49.477345 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:49.477346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:49.477504 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:49.477559 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.477561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.478172 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.478172 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.478679 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.478681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.478994 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.478995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.479057 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.479465 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:49.479469 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:49.479724 (+ 255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:49.481080 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:49.481081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:49.481082 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:49.481243 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:49.481405 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:49.481406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:49.481407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:49.481460 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:49.481705 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:49.481706 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:49.481749 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:46:49.481808 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:46:49.481901 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260620 14:46:49.482636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 8130
I20260620 14:46:49.482762 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:46:49.482947 392 external_mini_cluster.cc:1428] Setting key 7e9e84338c5af081e9192fb857f79918
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 14:46:49.504371 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41175
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:49.582316 8270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:49.582521 8270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:49.582543 8270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:49.583724 8270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:49.583767 8270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:49.583786 8270 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:46:49.583802 8270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:46:49.585279 8270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:49.585505 8270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:49.585714 8270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:49.590368 8278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:49.590402 8276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:49.590554 8275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:49.590576 8270 server_base.cc:1034] running on GCE node
I20260620 14:46:49.590735 8270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:49.590931 8270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:49.592079 8270 hybrid_clock.cc:648] HybridClock initialized: now 1781966809592067 us; error 30 us; skew 500 ppm
Jun 20 14:46:49 dist-test-slave-cx89 krb5kdc[7873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966809, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:49.594347 8270 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:46:49.594766 8270 webserver.cc:460] Webserver started at http://127.0.98.3:33187/ using document root <none> and password file <none>
I20260620 14:46:49.594969 8270 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:49.595013 8270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:49.595132 8270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:49.596176 8270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "3296214f8d0a44268b1458837df488c2"
format_stamp: "Formatted at 2026-06-20 14:46:49 on dist-test-slave-cx89"
server_key: "4423796bde7ff462b0f6f6e63209514f"
server_key_iv: "3175653f2c3010ae2fe722daa91c6df3"
server_key_version: "encryptionkey@0"
I20260620 14:46:49.596513 8270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3296214f8d0a44268b1458837df488c2"
format_stamp: "Formatted at 2026-06-20 14:46:49 on dist-test-slave-cx89"
server_key: "4423796bde7ff462b0f6f6e63209514f"
server_key_iv: "3175653f2c3010ae2fe722daa91c6df3"
server_key_version: "encryptionkey@0"
I20260620 14:46:49.597743 8270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:46:49.598345 8285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:49.598482 8270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:49.598557 8270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "3296214f8d0a44268b1458837df488c2"
format_stamp: "Formatted at 2026-06-20 14:46:49 on dist-test-slave-cx89"
server_key: "4423796bde7ff462b0f6f6e63209514f"
server_key_iv: "3175653f2c3010ae2fe722daa91c6df3"
server_key_version: "encryptionkey@0"
I20260620 14:46:49.598611 8270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:49.612677 8270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:49.614516 8270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:49.614650 8270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:49.615195 8270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:49.615244 8270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:49.615272 8270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:49.615303 8270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:46:49 dist-test-slave-cx89 krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966809, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:46:49.622146 8270 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40393
I20260620 14:46:49.622465 8270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:46:49.622872 8400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40393 every 8 connection(s)
I20260620 14:46:49.624725 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.615613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42157 (local address 127.0.98.62:41175)
0620 14:46:49.615678 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:49.615680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:49.615935 (+ 255us) server_negotiation.cc:407] Connection header received
0620 14:46:49.616583 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:49.616584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:49.616597 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:49.616669 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:49.617457 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.618000 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.619489 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.619715 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.619769 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.622137 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:49.622139 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:49.622139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:49.622147 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:49.623534 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:49.623879 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:49.623880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:49.623880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:49.623936 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:49.624146 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:49.624147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:49.624147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:49.624258 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:49.624313 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:49.624556 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 14:46:49.624640 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:46:49.624876 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.615592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41175 (local address 127.0.98.3:42157)
0620 14:46:49.615772 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:49.615829 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:49.616448 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:49.616746 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:49.616750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:49.617088 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:49.617376 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.617383 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.618083 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.618085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.619394 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.619397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.619790 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.619790 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.619945 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.620476 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:49.620485 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:49.622044 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:49.623610 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:49.623616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:49.623618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:49.623816 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:49.623987 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:49.623988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:49.623988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:49.624093 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:49.624310 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:49.624316 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:49.624445 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 14:46:49.624528 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:46:49.624672 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260620 14:46:49.625341 8401 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41175
I20260620 14:46:49.625424 8401 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:49.625604 8401 heartbeater.cc:511] Master 127.0.98.62:41175 requested a full tablet report, sending...
I20260620 14:46:49.625945 7917 ts_manager.cc:194] Registered new tserver with Master: 3296214f8d0a44268b1458837df488c2 (127.0.98.3:40393)
I20260620 14:46:49.626417 7917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42157
I20260620 14:46:49.628603 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 8270
I20260620 14:46:49.628700 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:46:49.628901 392 external_mini_cluster.cc:1428] Setting key 6e095341f455de489adcdccc18237b65
I20260620 14:46:49.629742 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:46:49.631706 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.626258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34067 (local address 127.0.98.62:41175)
0620 14:46:49.626305 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:49.626307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:49.626399 (+ 92us) server_negotiation.cc:407] Connection header received
0620 14:46:49.626459 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:49.626460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:49.626468 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:49.626524 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:49.626981 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.627520 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.628193 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.628376 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.628424 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.629298 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:49.629299 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:49.629299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:49.629306 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:49.630701 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:49.630986 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:49.630987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:49.630987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:49.631041 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:49.631165 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:49.631165 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:49.631172 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:49.631280 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:49.631336 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:49.631547 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:46:49.631620 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:46:49.631718 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.626204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41175 (local address 127.0.98.3:34067)
0620 14:46:49.626349 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:49.626358 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:49.626410 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:49.626545 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:49.626545 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:49.626828 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:49.626890 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.626892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.627601 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.627602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.628123 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.628124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.628427 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.628427 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.628470 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.628879 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:49.628885 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:49.629193 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:49.630779 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:49.630780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:49.630780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:49.630927 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:49.631091 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:49.631091 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:49.631091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:49.631132 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:49.631352 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:49.631352 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:49.631393 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:46:49.631512 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 14:46:49.631612 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":57,"threads_started":1}
Jun 20 14:46:49 dist-test-slave-cx89 krb5kdc[7873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966809, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:46:49 dist-test-slave-cx89 krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:49 dist-test-slave-cx89 krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:46:49.649720 7967 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:49.644792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60262 (local address 127.0.98.62:41175)
0620 14:46:49.644843 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:49.644844 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:49.644853 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:46:49.644929 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:49.644930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:49.644940 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:49.644985 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:49.645473 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.645950 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.646646 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.646828 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.646880 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.649627 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60262: BlockingRecv error: recv got EOF from 127.0.0.1:60262 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 14:46:49 dist-test-slave-cx89 krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966809, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:46:49.656440 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.650406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60264 (local address 127.0.98.62:41175)
0620 14:46:49.650463 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:49.650465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:49.650512 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:46:49.650551 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:49.650552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:49.650560 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:49.650604 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:49.651018 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.651524 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.652197 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.652379 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.652440 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.654322 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:49.654323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:49.654324 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:49.654330 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:49.655560 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:49.655860 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:49.655862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:49.655862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:49.655915 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:49.656072 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:49.656072 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:49.656073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:49.656171 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:49.656243 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:49.656313 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:46:49.656369 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:46:49.657848 7917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:46:49.659361 7917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:46:49.665457 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.661459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40393 (local address 127.0.0.1:36312)
0620 14:46:49.661959 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:49.661997 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:49.662058 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:49.662226 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:49.662229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:49.662232 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:46:49.662299 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.662301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.663224 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.663225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.664273 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.664275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.665155 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.665156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.665243 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.665259 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 14:46:49.665322 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:46:49.665373 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":72,"threads_started":1}
I20260620 14:46:49.665482 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.661555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.0.98.3:40393)
0620 14:46:49.661722 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:49.661727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:49.662047 (+ 320us) server_negotiation.cc:407] Connection header received
0620 14:46:49.662131 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:49.662133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:49.662145 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:49.662211 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:46:49.662362 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.663154 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.664366 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.665092 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.665146 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.665231 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:49.665339 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 14:46:49.665376 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260620 14:46:49.666134 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.661680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43177 (local address 127.0.0.1:60312)
0620 14:46:49.662225 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:49.662231 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:49.662268 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:49.662504 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:49.662505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:49.662507 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:46:49.662558 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.662559 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.663827 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.663828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.664942 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.664944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.665899 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.665899 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.665955 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.665958 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:46:49.666018 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:46:49.666058 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":29,"thread_start_us":45,"threads_started":1}
I20260620 14:46:49.666159 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.661787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.0.98.1:43177)
0620 14:46:49.661936 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:49.661939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:49.662318 (+ 379us) server_negotiation.cc:407] Connection header received
0620 14:46:49.662359 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:49.662362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:49.662376 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:49.662442 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:46:49.662623 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.663747 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.665021 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.665820 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.665899 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.665970 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:49.666033 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 14:46:49.666070 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260620 14:46:49.666244 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.661765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49424 (local address 127.0.98.2:45177)
0620 14:46:49.661934 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:49.661937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:49.662413 (+ 476us) server_negotiation.cc:407] Connection header received
0620 14:46:49.662456 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:49.662458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:49.662471 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:49.662559 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:46:49.662674 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.663753 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.664778 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.665398 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.665463 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.665530 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:49.666094 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 14:46:49.666140 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260620 14:46:49.666287 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.661473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45177 (local address 127.0.0.1:49424)
0620 14:46:49.662128 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:49.662134 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:49.662169 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:49.662571 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:49.662572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:49.662574 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:46:49.662622 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.662623 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.663838 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.663839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.664707 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.664709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.665564 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.665565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.665629 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.665633 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:46:49.665981 (+ 348us) client_negotiation.cc:240] Negotiation successful
0620 14:46:49.666028 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":35,"thread_start_us":63,"threads_started":1}
I20260620 14:46:49.666378 8335 tablet_service.cc:1467] Processing CreateTablet for tablet b3bbab606f92421cbb7693d3f3a0d437 (DEFAULT_TABLE table=test-table [id=34741b6c14a84bf78365ccbcc7bc5b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:49.666725 8335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3bbab606f92421cbb7693d3f3a0d437. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:49.666751 8055 tablet_service.cc:1467] Processing CreateTablet for tablet b3bbab606f92421cbb7693d3f3a0d437 (DEFAULT_TABLE table=test-table [id=34741b6c14a84bf78365ccbcc7bc5b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:49.666842 8195 tablet_service.cc:1467] Processing CreateTablet for tablet b3bbab606f92421cbb7693d3f3a0d437 (DEFAULT_TABLE table=test-table [id=34741b6c14a84bf78365ccbcc7bc5b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:49.667071 8055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3bbab606f92421cbb7693d3f3a0d437. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:49.667099 8195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3bbab606f92421cbb7693d3f3a0d437. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:49.669631 8426 tablet_bootstrap.cc:492] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2: Bootstrap starting.
I20260620 14:46:49.670223 8426 tablet_bootstrap.cc:654] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:49.670490 8426 log.cc:824] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:49.670996 8428 tablet_bootstrap.cc:492] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667: Bootstrap starting.
I20260620 14:46:49.671177 8426 tablet_bootstrap.cc:492] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2: No bootstrap required, opened a new log
I20260620 14:46:49.671234 8426 ts_tablet_manager.cc:1397] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:46:49.671571 8428 tablet_bootstrap.cc:654] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:49.671607 8427 tablet_bootstrap.cc:492] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4: Bootstrap starting.
I20260620 14:46:49.671909 8428 log.cc:824] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:49.672159 8427 tablet_bootstrap.cc:654] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:49.672446 8427 log.cc:824] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:49.672667 8428 tablet_bootstrap.cc:492] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667: No bootstrap required, opened a new log
I20260620 14:46:49.672737 8428 ts_tablet_manager.cc:1397] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:46:49.673123 8427 tablet_bootstrap.cc:492] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4: No bootstrap required, opened a new log
I20260620 14:46:49.673106 8426 raft_consensus.cc:348] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } } peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } }
I20260620 14:46:49.673240 8427 ts_tablet_manager.cc:1397] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:46:49.673270 8426 raft_consensus.cc:374] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:49.673309 8426 raft_consensus.cc:729] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3296214f8d0a44268b1458837df488c2, State: Initialized, Role: FOLLOWER
I20260620 14:46:49.673429 8426 consensus_queue.cc:260] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } } peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } }
I20260620 14:46:49.673722 8401 heartbeater.cc:503] Master 127.0.98.62:41175 was elected leader, sending a full tablet report...
I20260620 14:46:49.673975 8426 ts_tablet_manager.cc:1428] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:46:49.674688 8428 raft_consensus.cc:348] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } } peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } }
I20260620 14:46:49.674851 8428 raft_consensus.cc:374] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:49.674890 8428 raft_consensus.cc:729] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1978987151464256ba8d257d1ff48667, State: Initialized, Role: FOLLOWER
I20260620 14:46:49.675017 8428 consensus_queue.cc:260] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } } peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } }
I20260620 14:46:49.675288 8428 ts_tablet_manager.cc:1428] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:46:49.675323 8121 heartbeater.cc:503] Master 127.0.98.62:41175 was elected leader, sending a full tablet report...
I20260620 14:46:49.675621 8427 raft_consensus.cc:348] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } } peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } }
I20260620 14:46:49.675767 8427 raft_consensus.cc:374] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:49.675812 8427 raft_consensus.cc:729] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab203b9622e488c9001e049938329c4, State: Initialized, Role: FOLLOWER
I20260620 14:46:49.675940 8427 consensus_queue.cc:260] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } } peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } }
I20260620 14:46:49.676189 8427 ts_tablet_manager.cc:1428] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:46:49.676221 8261 heartbeater.cc:503] Master 127.0.98.62:41175 was elected leader, sending a full tablet report...
I20260620 14:46:49.689224 8434 raft_consensus.cc:482] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:46:49.689342 8434 raft_consensus.cc:504] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } } peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } }
I20260620 14:46:49.689656 8434 leader_election.cc:288] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3296214f8d0a44268b1458837df488c2 (127.0.98.3:40393), 1978987151464256ba8d257d1ff48667 (127.0.98.1:43177)
I20260620 14:46:49.693382 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.690032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33015 (local address 127.0.98.1:43177)
0620 14:46:49.690165 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:49.690167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:49.690173 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:46:49.690294 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:49.690295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:49.690306 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:49.690359 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:46:49.690479 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.691134 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.692484 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.693035 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.693104 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.693113 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:49.693277 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 14:46:49.693314 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260620 14:46:49.693393 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.689894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49155 (local address 127.0.98.3:40393)
0620 14:46:49.689946 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:49.689947 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:49.689977 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:46:49.690067 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:49.690068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:49.690079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:49.690122 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:46:49.690296 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.691037 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.692483 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.693048 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.693104 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.693113 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:49.693286 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 14:46:49.693329 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:46:49.693434 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.689960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43177 (local address 127.0.98.2:33015)
0620 14:46:49.690119 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:49.690129 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:49.690196 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:49.690368 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:49.690368 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:49.690370 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:46:49.690425 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.690426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.691193 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.691193 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.692388 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.692391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.693111 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.693112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.693205 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.693210 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:46:49.693265 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:46:49.693307 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":24,"thread_start_us":83,"threads_started":1}
I20260620 14:46:49.693449 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.689845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40393 (local address 127.0.98.2:49155)
0620 14:46:49.689923 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:49.689936 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:49.689995 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:49.690136 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:49.690137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:49.690139 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:46:49.690216 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.690218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.691110 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.691110 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.692388 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.692391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.693146 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.693148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.693218 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.693221 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:46:49.693268 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:46:49.693308 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":26}
I20260620 14:46:49.693778 8355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3bbab606f92421cbb7693d3f3a0d437" candidate_uuid: "bab203b9622e488c9001e049938329c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3296214f8d0a44268b1458837df488c2" is_pre_election: true
I20260620 14:46:49.693810 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3bbab606f92421cbb7693d3f3a0d437" candidate_uuid: "bab203b9622e488c9001e049938329c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1978987151464256ba8d257d1ff48667" is_pre_election: true
I20260620 14:46:49.693920 8075 raft_consensus.cc:2454] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bab203b9622e488c9001e049938329c4 in term 0.
I20260620 14:46:49.693928 8355 raft_consensus.cc:2454] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bab203b9622e488c9001e049938329c4 in term 0.
I20260620 14:46:49.694129 8148 leader_election.cc:302] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1978987151464256ba8d257d1ff48667, bab203b9622e488c9001e049938329c4; no voters:
I20260620 14:46:49.694244 8434 raft_consensus.cc:2781] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:46:49.694305 8434 raft_consensus.cc:482] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:46:49.694331 8434 raft_consensus.cc:3037] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:49.695272 8434 raft_consensus.cc:504] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } } peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } }
I20260620 14:46:49.695390 8434 leader_election.cc:288] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [CANDIDATE]: Term 1 election: Requested vote from peers 3296214f8d0a44268b1458837df488c2 (127.0.98.3:40393), 1978987151464256ba8d257d1ff48667 (127.0.98.1:43177)
I20260620 14:46:49.695539 8355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3bbab606f92421cbb7693d3f3a0d437" candidate_uuid: "bab203b9622e488c9001e049938329c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3296214f8d0a44268b1458837df488c2"
I20260620 14:46:49.695585 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3bbab606f92421cbb7693d3f3a0d437" candidate_uuid: "bab203b9622e488c9001e049938329c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1978987151464256ba8d257d1ff48667"
I20260620 14:46:49.695627 8355 raft_consensus.cc:3037] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:49.695642 8075 raft_consensus.cc:3037] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:49.696311 8355 raft_consensus.cc:2454] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bab203b9622e488c9001e049938329c4 in term 1.
I20260620 14:46:49.696329 8075 raft_consensus.cc:2454] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bab203b9622e488c9001e049938329c4 in term 1.
I20260620 14:46:49.696486 8148 leader_election.cc:302] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3296214f8d0a44268b1458837df488c2, bab203b9622e488c9001e049938329c4; no voters:
I20260620 14:46:49.696579 8434 raft_consensus.cc:2781] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:49.696735 8434 raft_consensus.cc:686] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [term 1 LEADER]: Becoming Leader. State: Replica: bab203b9622e488c9001e049938329c4, State: Running, Role: LEADER
I20260620 14:46:49.696822 8434 consensus_queue.cc:237] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } } peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } }
I20260620 14:46:49.697594 7917 catalog_manager.cc:5495] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to bab203b9622e488c9001e049938329c4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bab203b9622e488c9001e049938329c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } health_report { overall_health: HEALTHY } } }
I20260620 14:46:49.704516 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.702395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49440 (local address 127.0.98.2:45177)
0620 14:46:49.702433 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:49.702434 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:49.702456 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:46:49.702515 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:49.702515 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:49.702531 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:49.702583 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:46:49.702676 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.703124 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.703903 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.704087 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.704159 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.704270 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:46:49.704361 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:46:49.704412 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:49.704430 (+ 18us) server_negotiation.cc:299] Negotiation successful
0620 14:46:49.704447 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:46:49.706113 8075 raft_consensus.cc:1261] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667 [term 1 FOLLOWER]: Refusing update from remote peer bab203b9622e488c9001e049938329c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:49.706117 8355 raft_consensus.cc:1261] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2 [term 1 FOLLOWER]: Refusing update from remote peer bab203b9622e488c9001e049938329c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:49.706385 8434 consensus_queue.cc:1035] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:46:49.706516 8436 consensus_queue.cc:1035] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:46:49.708519 8439 mvcc.cc:205] Tried to move back new op lower bound from 7298936052553830400 to 7298936052518846464. Current Snapshot: MvccSnapshot[applied={T|T < 7298936052553830400}]
I20260620 14:46:49.709474 8441 mvcc.cc:205] Tried to move back new op lower bound from 7298936052553830400 to 7298936052518846464. Current Snapshot: MvccSnapshot[applied={T|T < 7298936052553830400 or (T in {7298936052553830400})}]
I20260620 14:46:49.710729 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:49.710810 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:49.711468 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299: Sending DeleteTablet for 3 replicas of tablet b3bbab606f92421cbb7693d3f3a0d437
I20260620 14:46:49.711756 8335 tablet_service.cc:1514] Processing DeleteTablet for tablet b3bbab606f92421cbb7693d3f3a0d437 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:49 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36312
I20260620 14:46:49.711861 8055 tablet_service.cc:1514] Processing DeleteTablet for tablet b3bbab606f92421cbb7693d3f3a0d437 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:49 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60312
I20260620 14:46:49.711961 8195 tablet_service.cc:1514] Processing DeleteTablet for tablet b3bbab606f92421cbb7693d3f3a0d437 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:49 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49424
I20260620 14:46:49.712098 8450 tablet_replica.cc:330] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667: stopping tablet replica
I20260620 14:46:49.712147 8449 tablet_replica.cc:330] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2: stopping tablet replica
I20260620 14:46:49.712160 7918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:46:49.712251 8449 raft_consensus.cc:2229] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 14:46:49.712294 7918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:46:49.712528 8450 raft_consensus.cc:2229] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:46:49.712594 8449 raft_consensus.cc:2258] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:49.712740 8450 raft_consensus.cc:2258] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:49.713138 8450 ts_tablet_manager.cc:1905] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:49.713506 8195 tablet_service.cc:1467] Processing CreateTablet for tablet ce004f1f2e314727bf8e1c257c261983 (DEFAULT_TABLE table=test-table [id=8b0981f4b50e4619913a61471d053071]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:49.713625 8195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce004f1f2e314727bf8e1c257c261983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:49.713701 8055 tablet_service.cc:1467] Processing CreateTablet for tablet ce004f1f2e314727bf8e1c257c261983 (DEFAULT_TABLE table=test-table [id=8b0981f4b50e4619913a61471d053071]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:49.713781 8055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce004f1f2e314727bf8e1c257c261983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:49.714888 8450 ts_tablet_manager.cc:1918] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:46:49.714949 8450 log.cc:1196] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/wal/wals/b3bbab606f92421cbb7693d3f3a0d437
I20260620 14:46:49.715078 8428 tablet_bootstrap.cc:492] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667: Bootstrap starting.
I20260620 14:46:49.715184 8450 ts_tablet_manager.cc:1939] T b3bbab606f92421cbb7693d3f3a0d437 P 1978987151464256ba8d257d1ff48667: Deleting consensus metadata
I20260620 14:46:49.715533 8427 tablet_bootstrap.cc:492] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4: Bootstrap starting.
I20260620 14:46:49.715590 8428 tablet_bootstrap.cc:654] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:49.715576 7904 catalog_manager.cc:4841] TS 1978987151464256ba8d257d1ff48667 (127.0.98.1:43177): tablet b3bbab606f92421cbb7693d3f3a0d437 (table test-table [id=34741b6c14a84bf78365ccbcc7bc5b4b]) successfully deleted
I20260620 14:46:49.715950 8427 tablet_bootstrap.cc:654] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:49.716234 8428 tablet_bootstrap.cc:492] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667: No bootstrap required, opened a new log
I20260620 14:46:49.716274 8428 ts_tablet_manager.cc:1397] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:49.716413 8428 raft_consensus.cc:348] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:49.716478 8428 raft_consensus.cc:374] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:49.716482 8427 tablet_bootstrap.cc:492] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4: No bootstrap required, opened a new log
I20260620 14:46:49.716495 8428 raft_consensus.cc:729] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1978987151464256ba8d257d1ff48667, State: Initialized, Role: FOLLOWER
I20260620 14:46:49.716523 8427 ts_tablet_manager.cc:1397] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:49.716531 8428 consensus_queue.cc:260] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:49.716607 8428 ts_tablet_manager.cc:1428] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:49.716727 8451 tablet_replica.cc:330] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4: stopping tablet replica
I20260620 14:46:49.716859 8451 raft_consensus.cc:2229] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:46:49.716666 8427 raft_consensus.cc:348] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:49.716953 8427 raft_consensus.cc:374] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:49.717000 8427 raft_consensus.cc:729] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab203b9622e488c9001e049938329c4, State: Initialized, Role: FOLLOWER
I20260620 14:46:49.717046 8427 consensus_queue.cc:260] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:49.717135 8451 raft_consensus.cc:2258] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:49.717144 8427 ts_tablet_manager.cc:1428] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:46:49.714989 8335 tablet_service.cc:1467] Processing CreateTablet for tablet ce004f1f2e314727bf8e1c257c261983 (DEFAULT_TABLE table=test-table [id=8b0981f4b50e4619913a61471d053071]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:49.717520 8335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce004f1f2e314727bf8e1c257c261983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:49.717651 8449 ts_tablet_manager.cc:1905] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:49.718756 8426 tablet_bootstrap.cc:492] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2: Bootstrap starting.
I20260620 14:46:49.718803 8449 ts_tablet_manager.cc:1918] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:46:49.718866 8449 log.cc:1196] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/b3bbab606f92421cbb7693d3f3a0d437
I20260620 14:46:49.719105 8449 ts_tablet_manager.cc:1939] T b3bbab606f92421cbb7693d3f3a0d437 P 3296214f8d0a44268b1458837df488c2: Deleting consensus metadata
I20260620 14:46:49.719162 8426 tablet_bootstrap.cc:654] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:49.719472 7904 catalog_manager.cc:4841] TS 3296214f8d0a44268b1458837df488c2 (127.0.98.3:40393): tablet b3bbab606f92421cbb7693d3f3a0d437 (table test-table [id=34741b6c14a84bf78365ccbcc7bc5b4b]) successfully deleted
I20260620 14:46:49.719720 8426 tablet_bootstrap.cc:492] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2: No bootstrap required, opened a new log
I20260620 14:46:49.719723 8451 ts_tablet_manager.cc:1905] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:49.719771 8426 ts_tablet_manager.cc:1397] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:49.719892 8426 raft_consensus.cc:348] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:49.719959 8426 raft_consensus.cc:374] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:49.719976 8426 raft_consensus.cc:729] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3296214f8d0a44268b1458837df488c2, State: Initialized, Role: FOLLOWER
I20260620 14:46:49.720013 8426 consensus_queue.cc:260] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:49.720091 8426 ts_tablet_manager.cc:1428] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:49.721112 8451 ts_tablet_manager.cc:1918] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:46:49.721170 8451 log.cc:1196] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/b3bbab606f92421cbb7693d3f3a0d437
I20260620 14:46:49.721396 8451 ts_tablet_manager.cc:1939] T b3bbab606f92421cbb7693d3f3a0d437 P bab203b9622e488c9001e049938329c4: Deleting consensus metadata
I20260620 14:46:49.721853 7905 catalog_manager.cc:4841] TS bab203b9622e488c9001e049938329c4 (127.0.98.2:45177): tablet b3bbab606f92421cbb7693d3f3a0d437 (table test-table [id=34741b6c14a84bf78365ccbcc7bc5b4b]) successfully deleted
W20260620 14:46:49.723450 8262 tablet.cc:2369] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:46:49.831378 8122 tablet.cc:2369] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:46:49.868815 8444 raft_consensus.cc:482] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:46:49.868914 8444 raft_consensus.cc:504] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:49.869052 8444 leader_election.cc:288] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1978987151464256ba8d257d1ff48667 (127.0.98.1:43177), 3296214f8d0a44268b1458837df488c2 (127.0.98.3:40393)
I20260620 14:46:49.869288 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce004f1f2e314727bf8e1c257c261983" candidate_uuid: "bab203b9622e488c9001e049938329c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1978987151464256ba8d257d1ff48667" is_pre_election: true
I20260620 14:46:49.869305 8355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce004f1f2e314727bf8e1c257c261983" candidate_uuid: "bab203b9622e488c9001e049938329c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3296214f8d0a44268b1458837df488c2" is_pre_election: true
I20260620 14:46:49.869382 8355 raft_consensus.cc:2454] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bab203b9622e488c9001e049938329c4 in term 0.
I20260620 14:46:49.869385 8075 raft_consensus.cc:2454] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bab203b9622e488c9001e049938329c4 in term 0.
I20260620 14:46:49.869550 8148 leader_election.cc:302] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3296214f8d0a44268b1458837df488c2, bab203b9622e488c9001e049938329c4; no voters:
I20260620 14:46:49.869652 8444 raft_consensus.cc:2781] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:46:49.869701 8444 raft_consensus.cc:482] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:46:49.869720 8444 raft_consensus.cc:3037] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:49.870390 8444 raft_consensus.cc:504] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:49.870491 8444 leader_election.cc:288] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [CANDIDATE]: Term 1 election: Requested vote from peers 1978987151464256ba8d257d1ff48667 (127.0.98.1:43177), 3296214f8d0a44268b1458837df488c2 (127.0.98.3:40393)
I20260620 14:46:49.870694 8355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce004f1f2e314727bf8e1c257c261983" candidate_uuid: "bab203b9622e488c9001e049938329c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3296214f8d0a44268b1458837df488c2"
I20260620 14:46:49.870698 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce004f1f2e314727bf8e1c257c261983" candidate_uuid: "bab203b9622e488c9001e049938329c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1978987151464256ba8d257d1ff48667"
I20260620 14:46:49.870759 8355 raft_consensus.cc:3037] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:49.870764 8075 raft_consensus.cc:3037] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:49.871321 8355 raft_consensus.cc:2454] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bab203b9622e488c9001e049938329c4 in term 1.
I20260620 14:46:49.871366 8075 raft_consensus.cc:2454] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bab203b9622e488c9001e049938329c4 in term 1.
I20260620 14:46:49.871497 8148 leader_election.cc:302] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3296214f8d0a44268b1458837df488c2, bab203b9622e488c9001e049938329c4; no voters:
I20260620 14:46:49.871587 8444 raft_consensus.cc:2781] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:49.871634 8444 raft_consensus.cc:686] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [term 1 LEADER]: Becoming Leader. State: Replica: bab203b9622e488c9001e049938329c4, State: Running, Role: LEADER
I20260620 14:46:49.871685 8444 consensus_queue.cc:237] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:49.872243 7918 catalog_manager.cc:5495] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to bab203b9622e488c9001e049938329c4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bab203b9622e488c9001e049938329c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } health_report { overall_health: UNKNOWN } } }
W20260620 14:46:49.872992 8402 tablet.cc:2369] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:46:49 dist-test-slave-cx89 krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966809, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:46:49.922258 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.916200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60274 (local address 127.0.98.62:41175)
0620 14:46:49.916254 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:49.916256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:49.916344 (+ 88us) server_negotiation.cc:407] Connection header received
0620 14:46:49.916403 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:49.916403 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:49.916412 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:49.916458 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:49.916883 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.917367 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.918007 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.918181 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.918232 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.920187 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:49.920188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:49.920189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:49.920195 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:49.921259 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:49.921572 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:49.921574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:49.921574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:49.921622 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:49.921772 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:49.921772 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:49.921773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:49.921880 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:49.921934 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:49.922067 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 14:46:49.922157 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:46:49.922331 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:49.916125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41175 (local address 127.0.0.1:60274)
0620 14:46:49.916281 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:49.916291 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:49.916356 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:49.916466 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:49.916466 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:49.916771 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:49.916814 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.916816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.917430 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.917431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.917943 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:49.917944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:49.918234 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:49.918235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:49.918279 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:49.918802 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:49.918807 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:49.920109 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:49.921327 (+ 1218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:49.921334 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:49.921337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:49.921511 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:49.921672 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:49.921673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:49.921673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:49.921723 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:49.921932 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:49.921938 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:49.921984 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 14:46:49.922040 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:46:49.922222 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":83,"thread_start_us":91,"threads_started":1}
I20260620 14:46:49.924165 7918 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60274:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:46:49.924389 7918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:46:49.925788 8055 tablet_service.cc:1467] Processing CreateTablet for tablet cdbbf774de234a45bf8dddb59503e869 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf06481307364857923ab6f0a65508de]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:46:49.925802 8195 tablet_service.cc:1467] Processing CreateTablet for tablet cdbbf774de234a45bf8dddb59503e869 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf06481307364857923ab6f0a65508de]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:46:49.925889 8335 tablet_service.cc:1467] Processing CreateTablet for tablet cdbbf774de234a45bf8dddb59503e869 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf06481307364857923ab6f0a65508de]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:46:49.925956 8195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdbbf774de234a45bf8dddb59503e869. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:49.925957 8055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdbbf774de234a45bf8dddb59503e869. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:49.926004 8335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdbbf774de234a45bf8dddb59503e869. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:49.927282 8428 tablet_bootstrap.cc:492] T cdbbf774de234a45bf8dddb59503e869 P 1978987151464256ba8d257d1ff48667: Bootstrap starting.
I20260620 14:46:49.927383 8426 tablet_bootstrap.cc:492] T cdbbf774de234a45bf8dddb59503e869 P 3296214f8d0a44268b1458837df488c2: Bootstrap starting.
I20260620 14:46:49.927737 8428 tablet_bootstrap.cc:654] T cdbbf774de234a45bf8dddb59503e869 P 1978987151464256ba8d257d1ff48667: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:49.927780 8426 tablet_bootstrap.cc:654] T cdbbf774de234a45bf8dddb59503e869 P 3296214f8d0a44268b1458837df488c2: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:49.928047 8427 tablet_bootstrap.cc:492] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4: Bootstrap starting.
I20260620 14:46:49.928344 8427 tablet_bootstrap.cc:654] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:49.928354 8428 tablet_bootstrap.cc:492] T cdbbf774de234a45bf8dddb59503e869 P 1978987151464256ba8d257d1ff48667: No bootstrap required, opened a new log
I20260620 14:46:49.928393 8426 tablet_bootstrap.cc:492] T cdbbf774de234a45bf8dddb59503e869 P 3296214f8d0a44268b1458837df488c2: No bootstrap required, opened a new log
I20260620 14:46:49.928398 8428 ts_tablet_manager.cc:1397] T cdbbf774de234a45bf8dddb59503e869 P 1978987151464256ba8d257d1ff48667: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:49.928435 8426 ts_tablet_manager.cc:1397] T cdbbf774de234a45bf8dddb59503e869 P 3296214f8d0a44268b1458837df488c2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:46:49.928546 8428 raft_consensus.cc:348] T cdbbf774de234a45bf8dddb59503e869 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:49.928579 8426 raft_consensus.cc:348] T cdbbf774de234a45bf8dddb59503e869 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:49.928642 8428 raft_consensus.cc:374] T cdbbf774de234a45bf8dddb59503e869 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:49.928668 8428 raft_consensus.cc:729] T cdbbf774de234a45bf8dddb59503e869 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1978987151464256ba8d257d1ff48667, State: Initialized, Role: FOLLOWER
I20260620 14:46:49.928670 8426 raft_consensus.cc:374] T cdbbf774de234a45bf8dddb59503e869 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:49.928694 8426 raft_consensus.cc:729] T cdbbf774de234a45bf8dddb59503e869 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3296214f8d0a44268b1458837df488c2, State: Initialized, Role: FOLLOWER
I20260620 14:46:49.928716 8428 consensus_queue.cc:260] T cdbbf774de234a45bf8dddb59503e869 P 1978987151464256ba8d257d1ff48667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:49.928731 8426 consensus_queue.cc:260] T cdbbf774de234a45bf8dddb59503e869 P 3296214f8d0a44268b1458837df488c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:49.928814 8426 ts_tablet_manager.cc:1428] T cdbbf774de234a45bf8dddb59503e869 P 3296214f8d0a44268b1458837df488c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:49.928825 8428 ts_tablet_manager.cc:1428] T cdbbf774de234a45bf8dddb59503e869 P 1978987151464256ba8d257d1ff48667: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:49.928875 8432 tablet_replica.cc:439] T cdbbf774de234a45bf8dddb59503e869 P 3296214f8d0a44268b1458837df488c2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } } }
I20260620 14:46:49.928963 8432 tablet_replica.cc:442] T cdbbf774de234a45bf8dddb59503e869 P 3296214f8d0a44268b1458837df488c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:46:49.929069 8433 tablet_replica.cc:439] T cdbbf774de234a45bf8dddb59503e869 P 1978987151464256ba8d257d1ff48667: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } } }
I20260620 14:46:49.929157 8433 tablet_replica.cc:442] T cdbbf774de234a45bf8dddb59503e869 P 1978987151464256ba8d257d1ff48667: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:46:49.929518 8427 tablet_bootstrap.cc:492] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4: No bootstrap required, opened a new log
I20260620 14:46:49.929570 8427 ts_tablet_manager.cc:1397] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:46:49.929670 8427 raft_consensus.cc:348] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:49.929742 8427 raft_consensus.cc:374] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:49.929759 8427 raft_consensus.cc:729] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab203b9622e488c9001e049938329c4, State: Initialized, Role: FOLLOWER
I20260620 14:46:49.929803 8427 consensus_queue.cc:260] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:49.929889 8427 ts_tablet_manager.cc:1428] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:49.929924 8444 tablet_replica.cc:439] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } } }
I20260620 14:46:49.930025 8444 tablet_replica.cc:442] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:46:50.052322 8444 raft_consensus.cc:482] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:46:50.052419 8444 raft_consensus.cc:504] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.052544 8444 leader_election.cc:288] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1978987151464256ba8d257d1ff48667 (127.0.98.1:43177), 3296214f8d0a44268b1458837df488c2 (127.0.98.3:40393)
I20260620 14:46:50.052776 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdbbf774de234a45bf8dddb59503e869" candidate_uuid: "bab203b9622e488c9001e049938329c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1978987151464256ba8d257d1ff48667" is_pre_election: true
I20260620 14:46:50.052834 8355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdbbf774de234a45bf8dddb59503e869" candidate_uuid: "bab203b9622e488c9001e049938329c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3296214f8d0a44268b1458837df488c2" is_pre_election: true
I20260620 14:46:50.052883 8075 raft_consensus.cc:2454] T cdbbf774de234a45bf8dddb59503e869 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bab203b9622e488c9001e049938329c4 in term 0.
I20260620 14:46:50.052908 8355 raft_consensus.cc:2454] T cdbbf774de234a45bf8dddb59503e869 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bab203b9622e488c9001e049938329c4 in term 0.
I20260620 14:46:50.053043 8148 leader_election.cc:302] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3296214f8d0a44268b1458837df488c2, bab203b9622e488c9001e049938329c4; no voters:
I20260620 14:46:50.053176 8444 raft_consensus.cc:2781] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:46:50.053220 8444 raft_consensus.cc:482] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:46:50.053231 8444 raft_consensus.cc:3037] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:50.053849 8444 raft_consensus.cc:504] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.053947 8444 leader_election.cc:288] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 [CANDIDATE]: Term 1 election: Requested vote from peers 1978987151464256ba8d257d1ff48667 (127.0.98.1:43177), 3296214f8d0a44268b1458837df488c2 (127.0.98.3:40393)
I20260620 14:46:50.054176 8355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdbbf774de234a45bf8dddb59503e869" candidate_uuid: "bab203b9622e488c9001e049938329c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3296214f8d0a44268b1458837df488c2"
I20260620 14:46:50.054176 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdbbf774de234a45bf8dddb59503e869" candidate_uuid: "bab203b9622e488c9001e049938329c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1978987151464256ba8d257d1ff48667"
I20260620 14:46:50.054275 8075 raft_consensus.cc:3037] T cdbbf774de234a45bf8dddb59503e869 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:50.054275 8355 raft_consensus.cc:3037] T cdbbf774de234a45bf8dddb59503e869 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:50.055025 8355 raft_consensus.cc:2454] T cdbbf774de234a45bf8dddb59503e869 P 3296214f8d0a44268b1458837df488c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bab203b9622e488c9001e049938329c4 in term 1.
I20260620 14:46:50.055025 8075 raft_consensus.cc:2454] T cdbbf774de234a45bf8dddb59503e869 P 1978987151464256ba8d257d1ff48667 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bab203b9622e488c9001e049938329c4 in term 1.
I20260620 14:46:50.055212 8148 leader_election.cc:302] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1978987151464256ba8d257d1ff48667, bab203b9622e488c9001e049938329c4; no voters:
I20260620 14:46:50.055311 8444 raft_consensus.cc:2781] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:50.055373 8444 raft_consensus.cc:686] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 [term 1 LEADER]: Becoming Leader. State: Replica: bab203b9622e488c9001e049938329c4, State: Running, Role: LEADER
I20260620 14:46:50.055446 8444 consensus_queue.cc:237] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.055738 8436 tablet_replica.cc:439] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4: TxnStatusTablet state changed. Reason: New leader bab203b9622e488c9001e049938329c4. Latest consensus state: current_term: 1 leader_uuid: "bab203b9622e488c9001e049938329c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } } }
I20260620 14:46:50.055821 8436 tablet_replica.cc:442] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:46:50.055953 8466 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:46:50.056068 8466 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:46:50.056092 7918 catalog_manager.cc:5495] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to bab203b9622e488c9001e049938329c4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bab203b9622e488c9001e049938329c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } health_report { overall_health: UNKNOWN } } }
I20260620 14:46:50.074724 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:50.072255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.0.98.2:45177)
0620 14:46:50.072295 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:50.072297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:50.072344 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:46:50.072438 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:50.072439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:50.072447 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:50.072492 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:46:50.072632 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.073101 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.074137 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.074316 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.074358 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:50.074529 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:46:50.074554 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:46:50.074584 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:50.074652 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 14:46:50.074666 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:46:50.074780 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:50.072211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45177 (local address 127.0.0.1:49456)
0620 14:46:50.072298 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:50.072309 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:50.072363 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:50.072503 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:50.072504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:50.072505 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:46:50.072570 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:50.072572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.073166 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.073167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:50.074059 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:50.074062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.074373 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.074374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:50.074435 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:50.074454 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:46:50.074588 (+ 134us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:46:50.074594 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:46:50.074649 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:46:50.074697 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 14:46:50.075682 8075 raft_consensus.cc:1261] T cdbbf774de234a45bf8dddb59503e869 P 1978987151464256ba8d257d1ff48667 [term 1 FOLLOWER]: Refusing update from remote peer bab203b9622e488c9001e049938329c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:50.075677 8355 raft_consensus.cc:1261] T cdbbf774de234a45bf8dddb59503e869 P 3296214f8d0a44268b1458837df488c2 [term 1 FOLLOWER]: Refusing update from remote peer bab203b9622e488c9001e049938329c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:50.075925 8444 consensus_queue.cc:1035] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:46:50.076054 8436 consensus_queue.cc:1035] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:46:50.076522 8433 tablet_replica.cc:439] T cdbbf774de234a45bf8dddb59503e869 P 1978987151464256ba8d257d1ff48667: TxnStatusTablet state changed. Reason: New leader bab203b9622e488c9001e049938329c4. Latest consensus state: current_term: 1 leader_uuid: "bab203b9622e488c9001e049938329c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } } }
I20260620 14:46:50.076514 8432 tablet_replica.cc:439] T cdbbf774de234a45bf8dddb59503e869 P 3296214f8d0a44268b1458837df488c2: TxnStatusTablet state changed. Reason: New leader bab203b9622e488c9001e049938329c4. Latest consensus state: current_term: 1 leader_uuid: "bab203b9622e488c9001e049938329c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } } }
I20260620 14:46:50.076624 8433 tablet_replica.cc:442] T cdbbf774de234a45bf8dddb59503e869 P 1978987151464256ba8d257d1ff48667: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:46:50.076628 8432 tablet_replica.cc:442] T cdbbf774de234a45bf8dddb59503e869 P 3296214f8d0a44268b1458837df488c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:46:50.077303 8434 tablet_replica.cc:439] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bab203b9622e488c9001e049938329c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } } }
I20260620 14:46:50.077378 8434 tablet_replica.cc:442] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:46:50.077418 8433 tablet_replica.cc:439] T cdbbf774de234a45bf8dddb59503e869 P 1978987151464256ba8d257d1ff48667: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bab203b9622e488c9001e049938329c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } } }
I20260620 14:46:50.077493 8433 tablet_replica.cc:442] T cdbbf774de234a45bf8dddb59503e869 P 1978987151464256ba8d257d1ff48667: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:46:50.078378 8436 tablet_replica.cc:439] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bab203b9622e488c9001e049938329c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } } }
I20260620 14:46:50.078465 8436 tablet_replica.cc:442] T cdbbf774de234a45bf8dddb59503e869 P bab203b9622e488c9001e049938329c4: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:46:50.078809 8432 tablet_replica.cc:439] T cdbbf774de234a45bf8dddb59503e869 P 3296214f8d0a44268b1458837df488c2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bab203b9622e488c9001e049938329c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } } }
I20260620 14:46:50.078894 8432 tablet_replica.cc:442] T cdbbf774de234a45bf8dddb59503e869 P 3296214f8d0a44268b1458837df488c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:46:50.082990 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:50.080834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37313 (local address 127.0.98.2:45177)
0620 14:46:50.080876 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:50.080878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:50.080950 (+ 72us) server_negotiation.cc:407] Connection header received
0620 14:46:50.081007 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:50.081007 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:50.081015 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:50.081057 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:46:50.081188 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.081643 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.082394 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.082584 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.082642 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:50.082781 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:46:50.082811 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:46:50.082847 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:50.082897 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 14:46:50.082915 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":23}
I20260620 14:46:50.083001 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:50.080747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45177 (local address 127.0.98.2:37313)
0620 14:46:50.080897 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:50.080907 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:50.080965 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:50.081058 (+ 93us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:50.081059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:50.081061 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:46:50.081138 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:50.081140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.081709 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.081709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:50.082325 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:50.082327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.082648 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.082649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:50.082709 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:50.082722 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:46:50.082858 (+ 136us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:46:50.082860 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:46:50.082896 (+ 36us) client_negotiation.cc:240] Negotiation successful
0620 14:46:50.082931 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":3,"thread_start_us":82,"threads_started":1}
I20260620 14:46:50.085165 8355 raft_consensus.cc:1261] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2 [term 1 FOLLOWER]: Refusing update from remote peer bab203b9622e488c9001e049938329c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:50.085201 8075 raft_consensus.cc:1261] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667 [term 1 FOLLOWER]: Refusing update from remote peer bab203b9622e488c9001e049938329c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:50.085369 8436 consensus_queue.cc:1035] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:46:50.085474 8444 consensus_queue.cc:1035] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:46:50.086823 8440 mvcc.cc:205] Tried to move back new op lower bound from 7298936054107787264 to 7298936053234716672. Current Snapshot: MvccSnapshot[applied={T|T < 7298936054107787264}]
I20260620 14:46:50.095331 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:50.095468 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:50.096071 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299: Sending DeleteTablet for 3 replicas of tablet ce004f1f2e314727bf8e1c257c261983
I20260620 14:46:50.096333 8055 tablet_service.cc:1514] Processing DeleteTablet for tablet ce004f1f2e314727bf8e1c257c261983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60312
I20260620 14:46:50.096441 8335 tablet_service.cc:1514] Processing DeleteTablet for tablet ce004f1f2e314727bf8e1c257c261983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36312
I20260620 14:46:50.096446 8450 tablet_replica.cc:330] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667: stopping tablet replica
I20260620 14:46:50.096510 8450 raft_consensus.cc:2229] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:46:50.096529 8449 tablet_replica.cc:330] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2: stopping tablet replica
I20260620 14:46:50.096545 8450 raft_consensus.cc:2258] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:50.096632 8449 raft_consensus.cc:2229] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:46:50.096702 8449 raft_consensus.cc:2258] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:50.096777 8195 tablet_service.cc:1514] Processing DeleteTablet for tablet ce004f1f2e314727bf8e1c257c261983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49424
I20260620 14:46:50.096838 8450 ts_tablet_manager.cc:1905] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:50.096851 8451 tablet_replica.cc:330] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4: stopping tablet replica
I20260620 14:46:50.096901 8451 raft_consensus.cc:2229] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:46:50.096943 8451 raft_consensus.cc:2258] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:50.097064 8449 ts_tablet_manager.cc:1905] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:50.097342 8451 ts_tablet_manager.cc:1905] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:50.098372 8450 ts_tablet_manager.cc:1918] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:46:50.098444 8450 log.cc:1196] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/wal/wals/ce004f1f2e314727bf8e1c257c261983
I20260620 14:46:50.098537 8449 ts_tablet_manager.cc:1918] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:46:50.098595 8449 log.cc:1196] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/ce004f1f2e314727bf8e1c257c261983
I20260620 14:46:50.098671 8450 ts_tablet_manager.cc:1939] T ce004f1f2e314727bf8e1c257c261983 P 1978987151464256ba8d257d1ff48667: Deleting consensus metadata
I20260620 14:46:50.098707 8451 ts_tablet_manager.cc:1918] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:46:50.098760 8451 log.cc:1196] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/ce004f1f2e314727bf8e1c257c261983
I20260620 14:46:50.098804 8449 ts_tablet_manager.cc:1939] T ce004f1f2e314727bf8e1c257c261983 P 3296214f8d0a44268b1458837df488c2: Deleting consensus metadata
I20260620 14:46:50.098935 8451 ts_tablet_manager.cc:1939] T ce004f1f2e314727bf8e1c257c261983 P bab203b9622e488c9001e049938329c4: Deleting consensus metadata
I20260620 14:46:50.098968 7904 catalog_manager.cc:4841] TS 1978987151464256ba8d257d1ff48667 (127.0.98.1:43177): tablet ce004f1f2e314727bf8e1c257c261983 (table test-table [id=8b0981f4b50e4619913a61471d053071]) successfully deleted
I20260620 14:46:50.099037 7904 catalog_manager.cc:4841] TS 3296214f8d0a44268b1458837df488c2 (127.0.98.3:40393): tablet ce004f1f2e314727bf8e1c257c261983 (table test-table [id=8b0981f4b50e4619913a61471d053071]) successfully deleted
I20260620 14:46:50.099205 7905 catalog_manager.cc:4841] TS bab203b9622e488c9001e049938329c4 (127.0.98.2:45177): tablet ce004f1f2e314727bf8e1c257c261983 (table test-table [id=8b0981f4b50e4619913a61471d053071]) successfully deleted
Jun 20 14:46:50 dist-test-slave-cx89 krb5kdc[7873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:46:50 dist-test-slave-cx89 krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:50 dist-test-slave-cx89 krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:46:50.120275 7967 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:50.114393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60278 (local address 127.0.98.62:41175)
0620 14:46:50.114449 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:50.114451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:50.114571 (+ 120us) server_negotiation.cc:407] Connection header received
0620 14:46:50.114640 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:50.114641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:50.114650 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:50.114703 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:50.115371 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.116003 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.117078 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.117284 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.117359 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:50.120209 (+ 2850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60278: BlockingRecv error: recv got EOF from 127.0.0.1:60278 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 14:46:50 dist-test-slave-cx89 krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:46:50.127307 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:50.121125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60288 (local address 127.0.98.62:41175)
0620 14:46:50.121175 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:50.121177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:50.121222 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:46:50.121284 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:50.121285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:50.121300 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:50.121343 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:50.121754 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.122213 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.122846 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.123031 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.123080 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:50.125002 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:50.125003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:50.125003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:50.125011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:50.126383 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:50.126672 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:50.126673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:50.126673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:50.126723 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:50.126875 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:50.126875 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:50.126876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:50.126973 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:50.127053 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:50.127154 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 14:46:50.127221 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:46:50.128690 7918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:46:50.128854 7918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:46:50.130192 8055 tablet_service.cc:1467] Processing CreateTablet for tablet bd4fed3c968e435faa80c4dad5f874c8 (DEFAULT_TABLE table=test-table [id=0fef26d91e294de98cd77d8d4f2fcbca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:50.130326 8055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4fed3c968e435faa80c4dad5f874c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:50.131760 8428 tablet_bootstrap.cc:492] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667: Bootstrap starting.
I20260620 14:46:50.131778 8195 tablet_service.cc:1467] Processing CreateTablet for tablet bd4fed3c968e435faa80c4dad5f874c8 (DEFAULT_TABLE table=test-table [id=0fef26d91e294de98cd77d8d4f2fcbca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:50.131887 8195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4fed3c968e435faa80c4dad5f874c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:50.132186 8428 tablet_bootstrap.cc:654] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:50.132772 8428 tablet_bootstrap.cc:492] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667: No bootstrap required, opened a new log
I20260620 14:46:50.132817 8428 ts_tablet_manager.cc:1397] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:50.132970 8428 raft_consensus.cc:348] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.133035 8428 raft_consensus.cc:374] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:50.133054 8428 raft_consensus.cc:729] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1978987151464256ba8d257d1ff48667, State: Initialized, Role: FOLLOWER
I20260620 14:46:50.133097 8428 consensus_queue.cc:260] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.133188 8335 tablet_service.cc:1467] Processing CreateTablet for tablet bd4fed3c968e435faa80c4dad5f874c8 (DEFAULT_TABLE table=test-table [id=0fef26d91e294de98cd77d8d4f2fcbca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:50.133220 8428 ts_tablet_manager.cc:1428] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:50.133301 8335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4fed3c968e435faa80c4dad5f874c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:50.134569 8427 tablet_bootstrap.cc:492] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4: Bootstrap starting.
I20260620 14:46:50.134852 8427 tablet_bootstrap.cc:654] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:50.135237 8426 tablet_bootstrap.cc:492] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2: Bootstrap starting.
I20260620 14:46:50.135622 8426 tablet_bootstrap.cc:654] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:50.136392 8427 tablet_bootstrap.cc:492] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4: No bootstrap required, opened a new log
I20260620 14:46:50.136442 8427 ts_tablet_manager.cc:1397] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:46:50.136544 8427 raft_consensus.cc:348] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.136607 8427 raft_consensus.cc:374] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:50.136626 8427 raft_consensus.cc:729] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab203b9622e488c9001e049938329c4, State: Initialized, Role: FOLLOWER
I20260620 14:46:50.136665 8427 consensus_queue.cc:260] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.136755 8427 ts_tablet_manager.cc:1428] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:50.137177 8426 tablet_bootstrap.cc:492] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2: No bootstrap required, opened a new log
I20260620 14:46:50.137229 8426 ts_tablet_manager.cc:1397] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:46:50.137368 8426 raft_consensus.cc:348] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.137454 8426 raft_consensus.cc:374] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:50.137475 8426 raft_consensus.cc:729] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3296214f8d0a44268b1458837df488c2, State: Initialized, Role: FOLLOWER
I20260620 14:46:50.137519 8426 consensus_queue.cc:260] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.137603 8426 ts_tablet_manager.cc:1428] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:50.434412 8432 raft_consensus.cc:482] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:46:50.434517 8432 raft_consensus.cc:504] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.434854 8432 leader_election.cc:288] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1978987151464256ba8d257d1ff48667 (127.0.98.1:43177), bab203b9622e488c9001e049938329c4 (127.0.98.2:45177)
I20260620 14:46:50.438582 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:50.435082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45177 (local address 127.0.98.3:57457)
0620 14:46:50.435294 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:50.435302 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:50.435352 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:50.435482 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:50.435482 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:50.435484 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:46:50.435560 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:50.435562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.436461 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.436463 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:50.437710 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:50.437713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.438374 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.438375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:50.438430 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:50.438434 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:46:50.438475 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 14:46:50.438508 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":8,"thread_start_us":78,"threads_started":1}
I20260620 14:46:50.438599 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:50.435121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57457 (local address 127.0.98.2:45177)
0620 14:46:50.435165 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:50.435168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:50.435360 (+ 192us) server_negotiation.cc:407] Connection header received
0620 14:46:50.435394 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:50.435395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:50.435409 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:50.435479 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:46:50.435622 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.436377 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.437796 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.438298 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.438373 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:50.438387 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:50.438488 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 14:46:50.438527 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:46:50.438818 8215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4fed3c968e435faa80c4dad5f874c8" candidate_uuid: "3296214f8d0a44268b1458837df488c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bab203b9622e488c9001e049938329c4" is_pre_election: true
I20260620 14:46:50.438954 8215 raft_consensus.cc:2454] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3296214f8d0a44268b1458837df488c2 in term 0.
I20260620 14:46:50.439061 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:50.435107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50907 (local address 127.0.98.1:43177)
0620 14:46:50.435239 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:50.435241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:50.435289 (+ 48us) server_negotiation.cc:407] Connection header received
0620 14:46:50.435378 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:50.435379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:50.435391 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:50.435480 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:46:50.435625 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.436375 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.438165 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.438725 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.438791 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:50.438806 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:50.438950 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 14:46:50.438988 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":56,"threads_started":1}
I20260620 14:46:50.439057 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:50.435069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43177 (local address 127.0.98.3:50907)
0620 14:46:50.435219 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:50.435230 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:50.435310 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:50.435483 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:50.435483 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:50.435485 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:46:50.435560 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:50.435562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.436463 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.436463 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:50.438055 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:50.438057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.438820 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.438821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:50.438886 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:50.438889 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:46:50.438934 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 14:46:50.438976 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":80,"thread_start_us":85,"threads_started":1}
I20260620 14:46:50.439145 8289 leader_election.cc:302] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3296214f8d0a44268b1458837df488c2, bab203b9622e488c9001e049938329c4; no voters:
I20260620 14:46:50.439289 8432 raft_consensus.cc:2781] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:46:50.439302 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4fed3c968e435faa80c4dad5f874c8" candidate_uuid: "3296214f8d0a44268b1458837df488c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1978987151464256ba8d257d1ff48667" is_pre_election: true
I20260620 14:46:50.439342 8432 raft_consensus.cc:482] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:46:50.439369 8432 raft_consensus.cc:3037] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:50.439376 8075 raft_consensus.cc:2454] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3296214f8d0a44268b1458837df488c2 in term 0.
I20260620 14:46:50.440084 8432 raft_consensus.cc:504] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.440222 8432 leader_election.cc:288] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [CANDIDATE]: Term 1 election: Requested vote from peers 1978987151464256ba8d257d1ff48667 (127.0.98.1:43177), bab203b9622e488c9001e049938329c4 (127.0.98.2:45177)
I20260620 14:46:50.440387 8215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4fed3c968e435faa80c4dad5f874c8" candidate_uuid: "3296214f8d0a44268b1458837df488c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bab203b9622e488c9001e049938329c4"
I20260620 14:46:50.440408 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4fed3c968e435faa80c4dad5f874c8" candidate_uuid: "3296214f8d0a44268b1458837df488c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1978987151464256ba8d257d1ff48667"
I20260620 14:46:50.440470 8215 raft_consensus.cc:3037] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:50.440485 8075 raft_consensus.cc:3037] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:50.441071 8075 raft_consensus.cc:2454] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3296214f8d0a44268b1458837df488c2 in term 1.
I20260620 14:46:50.441249 8215 raft_consensus.cc:2454] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3296214f8d0a44268b1458837df488c2 in term 1.
I20260620 14:46:50.441277 8288 leader_election.cc:302] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1978987151464256ba8d257d1ff48667, 3296214f8d0a44268b1458837df488c2; no voters:
I20260620 14:46:50.441346 8432 raft_consensus.cc:2781] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:50.441438 8432 raft_consensus.cc:686] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [term 1 LEADER]: Becoming Leader. State: Replica: 3296214f8d0a44268b1458837df488c2, State: Running, Role: LEADER
I20260620 14:46:50.441504 8432 consensus_queue.cc:237] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.442137 7918 catalog_manager.cc:5495] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3296214f8d0a44268b1458837df488c2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3296214f8d0a44268b1458837df488c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } health_report { overall_health: HEALTHY } } }
I20260620 14:46:50.491429 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:50.489323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36314 (local address 127.0.98.3:40393)
0620 14:46:50.489441 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:50.489443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:50.489450 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:46:50.489486 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:50.489487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:50.489506 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:50.489577 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:46:50.489662 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.490134 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.490834 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.491022 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.491064 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:50.491161 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:46:50.491263 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:46:50.491307 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:50.491327 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 14:46:50.491341 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
I20260620 14:46:50.492333 8075 raft_consensus.cc:1261] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667 [term 1 FOLLOWER]: Refusing update from remote peer 3296214f8d0a44268b1458837df488c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:50.492385 8215 raft_consensus.cc:1261] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4 [term 1 FOLLOWER]: Refusing update from remote peer 3296214f8d0a44268b1458837df488c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:50.492619 8495 consensus_queue.cc:1035] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:46:50.492745 8432 consensus_queue.cc:1035] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:46:50.496009 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:50.496111 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:50.496634 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299: Sending DeleteTablet for 3 replicas of tablet bd4fed3c968e435faa80c4dad5f874c8
I20260620 14:46:50.496842 8195 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4fed3c968e435faa80c4dad5f874c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49424
I20260620 14:46:50.496922 8451 tablet_replica.cc:330] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4: stopping tablet replica
I20260620 14:46:50.496898 8335 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4fed3c968e435faa80c4dad5f874c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36312
I20260620 14:46:50.496974 8451 raft_consensus.cc:2229] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:46:50.496981 8449 tablet_replica.cc:330] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2: stopping tablet replica
I20260620 14:46:50.497020 8449 raft_consensus.cc:2229] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:46:50.497027 8451 raft_consensus.cc:2258] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:50.497089 8449 raft_consensus.cc:2258] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:50.497088 7918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:46:50.497226 7918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:46:50.497323 8451 ts_tablet_manager.cc:1905] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:50.497403 8055 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4fed3c968e435faa80c4dad5f874c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60312
I20260620 14:46:50.497478 8450 tablet_replica.cc:330] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667: stopping tablet replica
I20260620 14:46:50.497534 8450 raft_consensus.cc:2229] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:46:50.497572 8450 raft_consensus.cc:2258] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:50.498071 8450 ts_tablet_manager.cc:1905] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:50.498070 8449 ts_tablet_manager.cc:1905] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:50.498741 8451 ts_tablet_manager.cc:1918] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:46:50.498795 8451 log.cc:1196] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/bd4fed3c968e435faa80c4dad5f874c8
I20260620 14:46:50.498991 8451 ts_tablet_manager.cc:1939] T bd4fed3c968e435faa80c4dad5f874c8 P bab203b9622e488c9001e049938329c4: Deleting consensus metadata
I20260620 14:46:50.499462 8449 ts_tablet_manager.cc:1918] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:46:50.499500 8450 ts_tablet_manager.cc:1918] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:46:50.499518 8449 log.cc:1196] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/bd4fed3c968e435faa80c4dad5f874c8
I20260620 14:46:50.499541 8450 log.cc:1196] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-0/wal/wals/bd4fed3c968e435faa80c4dad5f874c8
I20260620 14:46:50.499699 8449 ts_tablet_manager.cc:1939] T bd4fed3c968e435faa80c4dad5f874c8 P 3296214f8d0a44268b1458837df488c2: Deleting consensus metadata
I20260620 14:46:50.499729 8450 ts_tablet_manager.cc:1939] T bd4fed3c968e435faa80c4dad5f874c8 P 1978987151464256ba8d257d1ff48667: Deleting consensus metadata
I20260620 14:46:50.499724 7905 catalog_manager.cc:4841] TS bab203b9622e488c9001e049938329c4 (127.0.98.2:45177): tablet bd4fed3c968e435faa80c4dad5f874c8 (table test-table [id=0fef26d91e294de98cd77d8d4f2fcbca]) successfully deleted
I20260620 14:46:50.499782 8335 tablet_service.cc:1467] Processing CreateTablet for tablet 61c1720c2ac449bbb615706f42b133a8 (DEFAULT_TABLE table=test-table [id=7ba237b28d06495a943ababcef604ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:50.499874 8335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c1720c2ac449bbb615706f42b133a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:50.500854 8426 tablet_bootstrap.cc:492] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2: Bootstrap starting.
I20260620 14:46:50.500890 8195 tablet_service.cc:1467] Processing CreateTablet for tablet 61c1720c2ac449bbb615706f42b133a8 (DEFAULT_TABLE table=test-table [id=7ba237b28d06495a943ababcef604ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:50.500977 8195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c1720c2ac449bbb615706f42b133a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:50.501215 8426 tablet_bootstrap.cc:654] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:50.502058 8427 tablet_bootstrap.cc:492] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4: Bootstrap starting.
I20260620 14:46:50.502189 8055 tablet_service.cc:1467] Processing CreateTablet for tablet 61c1720c2ac449bbb615706f42b133a8 (DEFAULT_TABLE table=test-table [id=7ba237b28d06495a943ababcef604ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:46:50.502249 7904 catalog_manager.cc:4841] TS 3296214f8d0a44268b1458837df488c2 (127.0.98.3:40393): tablet bd4fed3c968e435faa80c4dad5f874c8 (table test-table [id=0fef26d91e294de98cd77d8d4f2fcbca]) successfully deleted
I20260620 14:46:50.502288 8055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c1720c2ac449bbb615706f42b133a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:50.502321 7904 catalog_manager.cc:4841] TS 1978987151464256ba8d257d1ff48667 (127.0.98.1:43177): tablet bd4fed3c968e435faa80c4dad5f874c8 (table test-table [id=0fef26d91e294de98cd77d8d4f2fcbca]) successfully deleted
I20260620 14:46:50.502341 8427 tablet_bootstrap.cc:654] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:50.502689 8426 tablet_bootstrap.cc:492] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2: No bootstrap required, opened a new log
I20260620 14:46:50.502718 8426 ts_tablet_manager.cc:1397] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:46:50.502827 8426 raft_consensus.cc:348] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.502876 8426 raft_consensus.cc:374] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:50.502889 8426 raft_consensus.cc:729] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3296214f8d0a44268b1458837df488c2, State: Initialized, Role: FOLLOWER
I20260620 14:46:50.502923 8426 consensus_queue.cc:260] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.502998 8426 ts_tablet_manager.cc:1428] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:50.503247 8427 tablet_bootstrap.cc:492] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4: No bootstrap required, opened a new log
I20260620 14:46:50.503284 8427 ts_tablet_manager.cc:1397] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:46:50.503361 8428 tablet_bootstrap.cc:492] T 61c1720c2ac449bbb615706f42b133a8 P 1978987151464256ba8d257d1ff48667: Bootstrap starting.
I20260620 14:46:50.503373 8427 raft_consensus.cc:348] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.503468 8427 raft_consensus.cc:374] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:50.503489 8427 raft_consensus.cc:729] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab203b9622e488c9001e049938329c4, State: Initialized, Role: FOLLOWER
I20260620 14:46:50.503525 8427 consensus_queue.cc:260] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.503598 8427 ts_tablet_manager.cc:1428] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:50.503758 8428 tablet_bootstrap.cc:654] T 61c1720c2ac449bbb615706f42b133a8 P 1978987151464256ba8d257d1ff48667: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:50.504369 8428 tablet_bootstrap.cc:492] T 61c1720c2ac449bbb615706f42b133a8 P 1978987151464256ba8d257d1ff48667: No bootstrap required, opened a new log
I20260620 14:46:50.504421 8428 ts_tablet_manager.cc:1397] T 61c1720c2ac449bbb615706f42b133a8 P 1978987151464256ba8d257d1ff48667: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:50.504554 8428 raft_consensus.cc:348] T 61c1720c2ac449bbb615706f42b133a8 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.504624 8428 raft_consensus.cc:374] T 61c1720c2ac449bbb615706f42b133a8 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:50.504650 8428 raft_consensus.cc:729] T 61c1720c2ac449bbb615706f42b133a8 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1978987151464256ba8d257d1ff48667, State: Initialized, Role: FOLLOWER
I20260620 14:46:50.504707 8428 consensus_queue.cc:260] T 61c1720c2ac449bbb615706f42b133a8 P 1978987151464256ba8d257d1ff48667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.504796 8428 ts_tablet_manager.cc:1428] T 61c1720c2ac449bbb615706f42b133a8 P 1978987151464256ba8d257d1ff48667: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:50.628064 8436 raft_consensus.cc:482] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:46:50.628175 8436 raft_consensus.cc:504] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.628307 8436 leader_election.cc:288] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1978987151464256ba8d257d1ff48667 (127.0.98.1:43177), 3296214f8d0a44268b1458837df488c2 (127.0.98.3:40393)
I20260620 14:46:50.628541 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c1720c2ac449bbb615706f42b133a8" candidate_uuid: "bab203b9622e488c9001e049938329c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3296214f8d0a44268b1458837df488c2" is_pre_election: true
I20260620 14:46:50.628542 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c1720c2ac449bbb615706f42b133a8" candidate_uuid: "bab203b9622e488c9001e049938329c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1978987151464256ba8d257d1ff48667" is_pre_election: true
I20260620 14:46:50.628638 8354 raft_consensus.cc:2454] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bab203b9622e488c9001e049938329c4 in term 0.
I20260620 14:46:50.628640 8075 raft_consensus.cc:2454] T 61c1720c2ac449bbb615706f42b133a8 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bab203b9622e488c9001e049938329c4 in term 0.
I20260620 14:46:50.628811 8148 leader_election.cc:302] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3296214f8d0a44268b1458837df488c2, bab203b9622e488c9001e049938329c4; no voters:
I20260620 14:46:50.628904 8436 raft_consensus.cc:2781] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:46:50.628938 8436 raft_consensus.cc:482] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:46:50.628970 8436 raft_consensus.cc:3037] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:50.629659 8436 raft_consensus.cc:504] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.629777 8436 leader_election.cc:288] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [CANDIDATE]: Term 1 election: Requested vote from peers 1978987151464256ba8d257d1ff48667 (127.0.98.1:43177), 3296214f8d0a44268b1458837df488c2 (127.0.98.3:40393)
I20260620 14:46:50.630023 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c1720c2ac449bbb615706f42b133a8" candidate_uuid: "bab203b9622e488c9001e049938329c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1978987151464256ba8d257d1ff48667"
I20260620 14:46:50.630024 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c1720c2ac449bbb615706f42b133a8" candidate_uuid: "bab203b9622e488c9001e049938329c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3296214f8d0a44268b1458837df488c2"
I20260620 14:46:50.630108 8354 raft_consensus.cc:3037] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:50.630115 8075 raft_consensus.cc:3037] T 61c1720c2ac449bbb615706f42b133a8 P 1978987151464256ba8d257d1ff48667 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:50.630836 8075 raft_consensus.cc:2454] T 61c1720c2ac449bbb615706f42b133a8 P 1978987151464256ba8d257d1ff48667 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bab203b9622e488c9001e049938329c4 in term 1.
I20260620 14:46:50.630836 8354 raft_consensus.cc:2454] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bab203b9622e488c9001e049938329c4 in term 1.
I20260620 14:46:50.631023 8148 leader_election.cc:302] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1978987151464256ba8d257d1ff48667, bab203b9622e488c9001e049938329c4; no voters:
I20260620 14:46:50.631119 8436 raft_consensus.cc:2781] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:50.631219 8436 raft_consensus.cc:686] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [term 1 LEADER]: Becoming Leader. State: Replica: bab203b9622e488c9001e049938329c4, State: Running, Role: LEADER
I20260620 14:46:50.631265 8436 consensus_queue.cc:237] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } }
I20260620 14:46:50.631824 7916 catalog_manager.cc:5495] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to bab203b9622e488c9001e049938329c4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bab203b9622e488c9001e049938329c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bab203b9622e488c9001e049938329c4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 } health_report { overall_health: UNKNOWN } } }
I20260620 14:46:50.651397 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:50.649581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49470 (local address 127.0.98.2:45177)
0620 14:46:50.649621 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:50.649623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:50.649630 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:46:50.649682 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:50.649683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:50.649691 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:50.649730 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:46:50.649811 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.650269 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.650916 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:50.651101 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:50.651147 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:50.651226 (+ 79us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:46:50.651251 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:46:50.651279 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:50.651313 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 14:46:50.651326 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:46:50.653400 8075 raft_consensus.cc:1261] T 61c1720c2ac449bbb615706f42b133a8 P 1978987151464256ba8d257d1ff48667 [term 1 FOLLOWER]: Refusing update from remote peer bab203b9622e488c9001e049938329c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:50.653400 8354 raft_consensus.cc:1261] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2 [term 1 FOLLOWER]: Refusing update from remote peer bab203b9622e488c9001e049938329c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:46:50.653580 8507 consensus_queue.cc:1035] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1978987151464256ba8d257d1ff48667" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:46:50.653685 8505 consensus_queue.cc:1035] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3296214f8d0a44268b1458837df488c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:46:50.662425 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:50.662508 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:46:50.663035 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 593202c7340441279dcc0d96ca78f299: Sending DeleteTablet for 3 replicas of tablet 61c1720c2ac449bbb615706f42b133a8
I20260620 14:46:50.663233 8195 tablet_service.cc:1514] Processing DeleteTablet for tablet 61c1720c2ac449bbb615706f42b133a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49424
I20260620 14:46:50.663233 8055 tablet_service.cc:1514] Processing DeleteTablet for tablet 61c1720c2ac449bbb615706f42b133a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60312
I20260620 14:46:50.663324 8335 tablet_service.cc:1514] Processing DeleteTablet for tablet 61c1720c2ac449bbb615706f42b133a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:46:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36312
I20260620 14:46:50.663434 8451 tablet_replica.cc:330] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4: stopping tablet replica
I20260620 14:46:50.663434 8449 tablet_replica.cc:330] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2: stopping tablet replica
I20260620 14:46:50.663488 8451 raft_consensus.cc:2229] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:46:50.663496 8449 raft_consensus.cc:2229] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:46:50.663525 8449 raft_consensus.cc:2258] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:50.663544 8451 raft_consensus.cc:2258] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:46:50.663573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 7990
I20260620 14:46:50.664701 8449 ts_tablet_manager.cc:1905] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:50.666189 8451 ts_tablet_manager.cc:1905] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:46:50.666216 8449 ts_tablet_manager.cc:1918] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:46:50.666263 8449 log.cc:1196] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/61c1720c2ac449bbb615706f42b133a8
I20260620 14:46:50.666443 8449 ts_tablet_manager.cc:1939] T 61c1720c2ac449bbb615706f42b133a8 P 3296214f8d0a44268b1458837df488c2: Deleting consensus metadata
I20260620 14:46:50.666695 7904 catalog_manager.cc:4841] TS 3296214f8d0a44268b1458837df488c2 (127.0.98.3:40393): tablet 61c1720c2ac449bbb615706f42b133a8 (table test-table [id=7ba237b28d06495a943ababcef604ef8]) successfully deleted
I20260620 14:46:50.667217 8451 ts_tablet_manager.cc:1918] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:46:50.667271 8451 log.cc:1196] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/61c1720c2ac449bbb615706f42b133a8
I20260620 14:46:50.667446 8451 ts_tablet_manager.cc:1939] T 61c1720c2ac449bbb615706f42b133a8 P bab203b9622e488c9001e049938329c4: Deleting consensus metadata
I20260620 14:46:50.667723 7905 catalog_manager.cc:4841] TS bab203b9622e488c9001e049938329c4 (127.0.98.2:45177): tablet 61c1720c2ac449bbb615706f42b133a8 (table test-table [id=7ba237b28d06495a943ababcef604ef8]) successfully deleted
W20260620 14:46:50.670296 7904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43177 (error 108)
I20260620 14:46:50.670786 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 8130
I20260620 14:46:50.670800 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:50.670637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43177 (local address 127.0.0.1:60318)
0620 14:46:50.670728 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:50.670766 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":47,"threads_started":1}
W20260620 14:46:50.670933 7904 catalog_manager.cc:4584] TS 1978987151464256ba8d257d1ff48667 (127.0.98.1:43177): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 61c1720c2ac449bbb615706f42b133a8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43177: connect: Connection refused (error 111)
I20260620 14:46:50.675802 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 8270
I20260620 14:46:50.680390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 7886
2026-06-20T14:46:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2728 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:46:50 dist-test-slave-cx89 krb5kdc[8517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:46:50 dist-test-slave-cx89 krb5kdc[8517](info): set up 2 sockets
Jun 20 14:46:50 dist-test-slave-cx89 krb5kdc[8517](info): commencing operation
krb5kdc: starting...
W20260620 14:46:51.631713 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:46:51 dist-test-slave-cx89 krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:46:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:46:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:46:51.739243 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37245
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:51.816253 8530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:51.816447 8530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:51.816466 8530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:51.817559 8530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:46:51.817591 8530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:51.817605 8530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:51.817615 8530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:46:51.817625 8530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:46:51.819044 8530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:51.819231 8530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:51.819393 8530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:51.823916 8536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:51.824062 8530 server_base.cc:1034] running on GCE node
W20260620 14:46:51.823911 8535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:51.823907 8538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:51.824366 8530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:51.824623 8530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:51.825759 8530 hybrid_clock.cc:648] HybridClock initialized: now 1781966811825751 us; error 40 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T14:46:51Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1146 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:46:51 dist-test-slave-cx89 krb5kdc[8543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:46:51 dist-test-slave-cx89 krb5kdc[8543](info): set up 2 sockets
Jun 20 14:46:51 dist-test-slave-cx89 krb5kdc[8543](info): commencing operation
krb5kdc: starting...
W20260620 14:46:52.795771 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:46:52 dist-test-slave-cx89 krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:46:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:46:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:46:52.908499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39179
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:52.986029 8556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:52.986244 8556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:52.986274 8556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:52.987388 8556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:46:52.987447 8556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:52.987468 8556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:52.987483 8556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:46:52.987504 8556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:46:52.988974 8556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:52.989216 8556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:52.989406 8556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:52.993842 8561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:52.993836 8562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:52.993845 8564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:52.993965 8556 server_base.cc:1034] running on GCE node
I20260620 14:46:52.994239 8556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:52.994522 8556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:52.995652 8556 hybrid_clock.cc:648] HybridClock initialized: now 1781966812995640 us; error 27 us; skew 500 ppm
Jun 20 14:46:52 dist-test-slave-cx89 krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:52.997884 8556 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:46:52.998292 8556 webserver.cc:460] Webserver started at http://127.0.98.62:44961/ using document root <none> and password file <none>
I20260620 14:46:52.998495 8556 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:52.998536 8556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:52.998652 8556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:52.999670 8556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "ac6e149f076a43cca9997613200965ae"
format_stamp: "Formatted at 2026-06-20 14:46:52 on dist-test-slave-cx89"
server_key: "dddca2d2a0897f6e1e7fbeeff674725f"
server_key_iv: "8b51fbd06ef4cfd8003ad4f514a076c7"
server_key_version: "encryptionkey@0"
I20260620 14:46:52.999989 8556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "ac6e149f076a43cca9997613200965ae"
format_stamp: "Formatted at 2026-06-20 14:46:52 on dist-test-slave-cx89"
server_key: "dddca2d2a0897f6e1e7fbeeff674725f"
server_key_iv: "8b51fbd06ef4cfd8003ad4f514a076c7"
server_key_version: "encryptionkey@0"
I20260620 14:46:53.001169 8556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:53.001832 8571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:53.001959 8556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:53.002038 8556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "ac6e149f076a43cca9997613200965ae"
format_stamp: "Formatted at 2026-06-20 14:46:52 on dist-test-slave-cx89"
server_key: "dddca2d2a0897f6e1e7fbeeff674725f"
server_key_iv: "8b51fbd06ef4cfd8003ad4f514a076c7"
server_key_version: "encryptionkey@0"
I20260620 14:46:53.002094 8556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:53.017982 8556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:53.019819 8556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:53.019959 8556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:53.023865 8556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39179
I20260620 14:46:53.023893 8623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39179 every 8 connection(s)
I20260620 14:46:53.024363 8556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:46:53.025005 8624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:46:53.027238 8624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae: Bootstrap starting.
I20260620 14:46:53.027865 8624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae: Neither blocks nor log segments found. Creating new log.
I20260620 14:46:53.028102 8624 log.cc:824] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:53.028730 8624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae: No bootstrap required, opened a new log
I20260620 14:46:53.030273 8624 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6e149f076a43cca9997613200965ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39179 } }
I20260620 14:46:53.030414 8624 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:46:53.030445 8624 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6e149f076a43cca9997613200965ae, State: Initialized, Role: FOLLOWER
I20260620 14:46:53.030565 8624 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6e149f076a43cca9997613200965ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39179 } }
I20260620 14:46:53.030646 8624 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:46:53.030675 8624 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:46:53.030705 8624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:46:53.031549 8624 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6e149f076a43cca9997613200965ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39179 } }
I20260620 14:46:53.031636 8624 leader_election.cc:302] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac6e149f076a43cca9997613200965ae; no voters:
I20260620 14:46:53.031764 8624 leader_election.cc:288] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:46:53.031826 8627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:46:53.031992 8624 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:46:53.031992 8627 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [term 1 LEADER]: Becoming Leader. State: Replica: ac6e149f076a43cca9997613200965ae, State: Running, Role: LEADER
I20260620 14:46:53.032158 8627 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6e149f076a43cca9997613200965ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39179 } }
I20260620 14:46:53.032510 8629 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac6e149f076a43cca9997613200965ae. Latest consensus state: current_term: 1 leader_uuid: "ac6e149f076a43cca9997613200965ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6e149f076a43cca9997613200965ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39179 } } }
I20260620 14:46:53.032853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 8556
I20260620 14:46:53.032959 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:46:53.033171 392 external_mini_cluster.cc:1428] Setting key f7f688f88aa35544345594c5dc5e5875
I20260620 14:46:53.033509 8629 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:53.033630 8628 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac6e149f076a43cca9997613200965ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6e149f076a43cca9997613200965ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39179 } } }
I20260620 14:46:53.033747 8628 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [sys.catalog]: This master's current role is: LEADER
W20260620 14:46:53.033890 8642 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:46:53.033944 8642 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:46:53.035547 8645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:46:53.035730 8645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:46:53.037309 8645 catalog_manager.cc:1347] Generated new cluster ID: f21d3152f2014c32a7706bdf05a24050
I20260620 14:46:53.037353 8645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:46:53 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:53.044057 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:53.033798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37012 (local address 127.0.98.62:39179)
0620 14:46:53.033941 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:53.033951 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:53.034142 (+ 191us) server_negotiation.cc:407] Connection header received
0620 14:46:53.034262 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:53.034268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:53.034311 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:53.034420 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:53.034992 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.036069 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.037122 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.037714 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.037810 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:53.040596 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:53.040602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:53.040602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:53.040614 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:53.042503 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:53.042868 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:53.042871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:53.042871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:53.042978 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:53.043151 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:53.043152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:53.043152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:53.043495 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:53.043641 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:53.043715 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 14:46:53.043851 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":69,"threads_started":1}
I20260620 14:46:53.063192 8645 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:46:53.063771 8645 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:46:53.072527 8645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae: Generated new TSK 0
I20260620 14:46:53.072710 8645 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:46:53.098804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39179
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:53.181252 8650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:53.181535 8650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:53.181571 8650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:53.183033 8650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:53.183090 8650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:53.183130 8650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:46:53.184830 8650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:53.185019 8650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:53.185196 8650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:53.189744 8655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:53.189862 8650 server_base.cc:1034] running on GCE node
W20260620 14:46:53.189710 8658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:53.189710 8656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:53.190122 8650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:53.190333 8650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:53.191465 8650 hybrid_clock.cc:648] HybridClock initialized: now 1781966813191453 us; error 30 us; skew 500 ppm
Jun 20 14:46:53 dist-test-slave-cx89 krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:53.193787 8650 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:46:53.194180 8650 webserver.cc:460] Webserver started at http://127.0.98.1:33375/ using document root <none> and password file <none>
I20260620 14:46:53.194367 8650 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:53.194413 8650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:53.194530 8650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:53.195560 8650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "6724709e381c4d33b9113b68fc7ddc5e"
format_stamp: "Formatted at 2026-06-20 14:46:53 on dist-test-slave-cx89"
server_key: "c4c829fb5be3fe9deb05e50675b93aa4"
server_key_iv: "96bb0afd853c3cdfb25b917b99da7a40"
server_key_version: "encryptionkey@0"
I20260620 14:46:53.195870 8650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6724709e381c4d33b9113b68fc7ddc5e"
format_stamp: "Formatted at 2026-06-20 14:46:53 on dist-test-slave-cx89"
server_key: "c4c829fb5be3fe9deb05e50675b93aa4"
server_key_iv: "96bb0afd853c3cdfb25b917b99da7a40"
server_key_version: "encryptionkey@0"
I20260620 14:46:53.197007 8650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:53.197598 8665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:53.197746 8650 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:53.197820 8650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "6724709e381c4d33b9113b68fc7ddc5e"
format_stamp: "Formatted at 2026-06-20 14:46:53 on dist-test-slave-cx89"
server_key: "c4c829fb5be3fe9deb05e50675b93aa4"
server_key_iv: "96bb0afd853c3cdfb25b917b99da7a40"
server_key_version: "encryptionkey@0"
I20260620 14:46:53.197885 8650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:53.210322 8650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:53.212304 8650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:53.212433 8650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:53.212628 8650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:53.212909 8650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:53.212944 8650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:53.212971 8650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:53.212993 8650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:53.218200 8650 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37241
I20260620 14:46:53.218240 8778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37241 every 8 connection(s)
I20260620 14:46:53.218521 8650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:46:53.223856 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 8650
I20260620 14:46:53.223946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:46:53.224126 392 external_mini_cluster.cc:1428] Setting key eee203d171c9d4b7c12fcf2c5f93108e
Jun 20 14:46:53 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:53.227380 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:53.219164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58575 (local address 127.0.98.62:39179)
0620 14:46:53.219215 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:53.219217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:53.219404 (+ 187us) server_negotiation.cc:407] Connection header received
0620 14:46:53.220326 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:53.220328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:53.220342 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:53.220401 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:53.220937 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.221430 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.222359 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.222570 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.222640 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:53.224814 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:53.224815 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:53.224816 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:53.224823 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:53.226044 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:53.226522 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:53.226522 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:53.226523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:53.226568 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:53.226802 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:53.226803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:53.226803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:53.226900 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:53.226942 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:53.227254 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 14:46:53.227314 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:46:53.227659 8781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:53.219147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:58575)
0620 14:46:53.219290 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:53.219329 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:53.220228 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:53.220454 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:53.220457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:53.220680 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:53.220861 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:53.220865 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.221494 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.221494 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:53.222268 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:53.222270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.222672 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.222673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:53.222784 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:53.223277 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:53.223286 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:53.224730 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:53.226121 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:53.226128 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:53.226131 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:53.226438 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:53.226609 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:53.226610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:53.226610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:53.226742 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:53.226953 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:53.226958 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:53.227116 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 14:46:53.227239 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 14:46:53.227399 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260620 14:46:53.228295 8779 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39179
I20260620 14:46:53.228389 8779 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:53.228559 8779 heartbeater.cc:511] Master 127.0.98.62:39179 requested a full tablet report, sending...
I20260620 14:46:53.229033 8588 ts_manager.cc:194] Registered new tserver with Master: 6724709e381c4d33b9113b68fc7ddc5e (127.0.98.1:37241)
I20260620 14:46:53.229614 8588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58575
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:46:53.247090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39179
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:53.323207 8784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:53.323433 8784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:53.323465 8784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:53.324610 8784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:53.324651 8784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:53.324685 8784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:46:53.326135 8784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:53.326365 8784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:53.326560 8784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:53.331615 8790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:53.331605 8789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:53.331601 8792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:53.331642 8784 server_base.cc:1034] running on GCE node
I20260620 14:46:53.331995 8784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:53.332232 8784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:53.333410 8784 hybrid_clock.cc:648] HybridClock initialized: now 1781966813333394 us; error 25 us; skew 500 ppm
Jun 20 14:46:53 dist-test-slave-cx89 krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:53.335909 8784 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:46:53.336391 8784 webserver.cc:460] Webserver started at http://127.0.98.2:36065/ using document root <none> and password file <none>
I20260620 14:46:53.336589 8784 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:53.336639 8784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:53.336771 8784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:53.337874 8784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "af4c554f688b44b194382fd537248b8e"
format_stamp: "Formatted at 2026-06-20 14:46:53 on dist-test-slave-cx89"
server_key: "709d915a27c448ac065527fd5178a006"
server_key_iv: "4ee00fe8efea683a3bde18e1830e0746"
server_key_version: "encryptionkey@0"
I20260620 14:46:53.338225 8784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "af4c554f688b44b194382fd537248b8e"
format_stamp: "Formatted at 2026-06-20 14:46:53 on dist-test-slave-cx89"
server_key: "709d915a27c448ac065527fd5178a006"
server_key_iv: "4ee00fe8efea683a3bde18e1830e0746"
server_key_version: "encryptionkey@0"
I20260620 14:46:53.339501 8784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:46:53.340189 8799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:53.340346 8784 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:53.340430 8784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "af4c554f688b44b194382fd537248b8e"
format_stamp: "Formatted at 2026-06-20 14:46:53 on dist-test-slave-cx89"
server_key: "709d915a27c448ac065527fd5178a006"
server_key_iv: "4ee00fe8efea683a3bde18e1830e0746"
server_key_version: "encryptionkey@0"
I20260620 14:46:53.340492 8784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:53.353799 8784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:53.355952 8784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:53.356091 8784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:53.356302 8784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:53.356590 8784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:53.356626 8784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:53.356657 8784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:53.356678 8784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:53.362541 8784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43195
I20260620 14:46:53.362579 8912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43195 every 8 connection(s)
I20260620 14:46:53.362870 8784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:46:53 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:53.370976 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:53.363560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41733 (local address 127.0.98.62:39179)
0620 14:46:53.363616 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:53.363618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:53.363804 (+ 186us) server_negotiation.cc:407] Connection header received
0620 14:46:53.364688 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:53.364689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:53.364700 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:53.364762 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:53.365316 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.365813 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.366701 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.366877 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.366927 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:53.368809 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:53.368811 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:53.368812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:53.368818 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:53.370025 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:53.370315 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:53.370316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:53.370316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:53.370363 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:53.370521 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:53.370522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:53.370522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:53.370612 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:53.370655 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:53.370855 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 14:46:53.370914 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:46:53.371063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 8784
I20260620 14:46:53.371152 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:46:53.371177 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:53.363532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:41733)
0620 14:46:53.363678 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:53.363718 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:53.364590 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:53.364824 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:53.364828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:53.365065 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:53.365251 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:53.365255 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.365875 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.365876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:53.366638 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:53.366640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.366932 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.366932 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:53.367021 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:53.367458 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:53.367468 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:53.368722 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:53.370087 (+ 1365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:53.370090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:53.370091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:53.370269 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:53.370405 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:53.370405 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:53.370406 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:53.370482 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:53.370651 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:53.370654 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:53.370760 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 14:46:53.370832 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:46:53.370966 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260620 14:46:53.371361 392 external_mini_cluster.cc:1428] Setting key 5ab7bb700dee62862c7f0dd77b528a2c
I20260620 14:46:53.371587 8913 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39179
I20260620 14:46:53.371676 8913 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:53.371805 8913 heartbeater.cc:511] Master 127.0.98.62:39179 requested a full tablet report, sending...
I20260620 14:46:53.373236 8588 ts_manager.cc:194] Registered new tserver with Master: af4c554f688b44b194382fd537248b8e (127.0.98.2:43195)
I20260620 14:46:53.373586 8588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41733
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:46:53.394774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39179
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:53.473533 8918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:53.473737 8918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:53.473758 8918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:53.474879 8918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:53.474920 8918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:53.474946 8918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:46:53.476476 8918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:53.476693 8918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:53.476899 8918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:53.481509 8923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:53.481560 8924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:53.481523 8926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:53.481712 8918 server_base.cc:1034] running on GCE node
I20260620 14:46:53.481951 8918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:53.482182 8918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:53.483315 8918 hybrid_clock.cc:648] HybridClock initialized: now 1781966813483305 us; error 33 us; skew 500 ppm
Jun 20 14:46:53 dist-test-slave-cx89 krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:53.485652 8918 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:46:53.486081 8918 webserver.cc:460] Webserver started at http://127.0.98.3:45281/ using document root <none> and password file <none>
I20260620 14:46:53.486272 8918 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:53.486320 8918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:53.486434 8918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:46:53.487583 8918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "273c591cfcbb4fcd83a2be8a97958326"
format_stamp: "Formatted at 2026-06-20 14:46:53 on dist-test-slave-cx89"
server_key: "03e3ab58f7bcddcf321033c54e9a0122"
server_key_iv: "93e1255588800b5a3bce014cbce3f2ac"
server_key_version: "encryptionkey@0"
I20260620 14:46:53.487914 8918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "273c591cfcbb4fcd83a2be8a97958326"
format_stamp: "Formatted at 2026-06-20 14:46:53 on dist-test-slave-cx89"
server_key: "03e3ab58f7bcddcf321033c54e9a0122"
server_key_iv: "93e1255588800b5a3bce014cbce3f2ac"
server_key_version: "encryptionkey@0"
I20260620 14:46:53.489090 8918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:53.489743 8933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:53.489890 8918 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:53.489967 8918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "273c591cfcbb4fcd83a2be8a97958326"
format_stamp: "Formatted at 2026-06-20 14:46:53 on dist-test-slave-cx89"
server_key: "03e3ab58f7bcddcf321033c54e9a0122"
server_key_iv: "93e1255588800b5a3bce014cbce3f2ac"
server_key_version: "encryptionkey@0"
I20260620 14:46:53.490026 8918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:53.507007 8918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:53.508816 8918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:53.508944 8918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:53.509142 8918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:53.509418 8918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:53.509449 8918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:53.509476 8918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:53.509493 8918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:53.514788 8918 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42355
I20260620 14:46:53.514844 9046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42355 every 8 connection(s)
I20260620 14:46:53.515120 8918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:46:53.519291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 8918
I20260620 14:46:53.519375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:46:53.519583 392 external_mini_cluster.cc:1428] Setting key 29c98172dd96f7e5183a19ef64b02b08
Jun 20 14:46:53 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:46:53.524753 9049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:53.515729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.3:42315)
0620 14:46:53.515882 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:53.515933 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:53.516795 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:53.516993 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:53.516997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:53.517232 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:53.517423 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:53.517427 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.518032 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.518033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:53.518820 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:53.518822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.519122 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.519123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:53.519211 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:53.519756 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:53.519767 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:53.521198 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:46:53.522551 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:53.522559 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:53.522561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:53.523072 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:53.523326 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:46:53.523327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:46:53.523327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:46:53.523507 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:46:53.523768 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:46:53.523774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:46:53.524029 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 14:46:53.524150 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 14:46:53.524362 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260620 14:46:53.526279 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:53.515765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42315 (local address 127.0.98.62:39179)
0620 14:46:53.515823 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:53.515825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:53.516008 (+ 183us) server_negotiation.cc:407] Connection header received
0620 14:46:53.516877 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:53.516878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:53.516887 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:53.516940 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:53.517481 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.517960 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.518880 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.519054 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.519101 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:53.521301 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:53.521302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:53.521302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:53.521308 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:53.522455 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:53.523155 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:53.523157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:53.523157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:53.523225 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:53.523562 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:53.523563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:53.523563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:53.523710 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:53.523768 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:53.526134 (+ 2366us) server_negotiation.cc:299] Negotiation successful
0620 14:46:53.526212 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:46:53.527781 9047 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39179
I20260620 14:46:53.527922 9047 heartbeater.cc:464] Registering TS with master...
I20260620 14:46:53.528126 9047 heartbeater.cc:511] Master 127.0.98.62:39179 requested a full tablet report, sending...
I20260620 14:46:53.528752 8588 ts_manager.cc:194] Registered new tserver with Master: 273c591cfcbb4fcd83a2be8a97958326 (127.0.98.3:42355)
I20260620 14:46:53.529307 8588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42315
I20260620 14:46:53.531307 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 14:46:53.603646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 8650
I20260620 14:46:53.607506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 8784
I20260620 14:46:53.610997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 8918
I20260620 14:46:53.614435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 8556
I20260620 14:46:53.618745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39179
--webserver_interface=127.0.98.62
--webserver_port=44961
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:46:53.697781 9056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:53.697991 9056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:53.698014 9056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:53.699108 9056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:46:53.699143 9056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:53.699158 9056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:53.699172 9056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:46:53.699183 9056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:46:53.700652 9056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:53.700857 9056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:53.701030 9056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:53.705595 9061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:53.705715 9056 server_base.cc:1034] running on GCE node
W20260620 14:46:53.705638 9062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:53.705822 9064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:53.706049 9056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:53.706331 9056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:53.707485 9056 hybrid_clock.cc:648] HybridClock initialized: now 1781966813707474 us; error 27 us; skew 500 ppm
Jun 20 14:46:53 dist-test-slave-cx89 krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:53.709852 9056 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:46:53.710269 9056 webserver.cc:460] Webserver started at http://127.0.98.62:44961/ using document root <none> and password file <none>
I20260620 14:46:53.710474 9056 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:53.710529 9056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:53.711963 9056 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:53.712559 9071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:53.712697 9056 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:53.712781 9056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "ac6e149f076a43cca9997613200965ae"
format_stamp: "Formatted at 2026-06-20 14:46:52 on dist-test-slave-cx89"
server_key: "dddca2d2a0897f6e1e7fbeeff674725f"
server_key_iv: "8b51fbd06ef4cfd8003ad4f514a076c7"
server_key_version: "encryptionkey@0"
I20260620 14:46:53.713121 9056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:53.732722 9056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:53.734834 9056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:53.734969 9056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:53.738801 9056 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39179
I20260620 14:46:53.738839 9123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39179 every 8 connection(s)
I20260620 14:46:53.739327 9056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:46:53.740511 9124 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:46:53.741107 9124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae: Bootstrap starting.
I20260620 14:46:53.743252 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 9056
I20260620 14:46:53.743556 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37241
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39179
--webserver_port=33375
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:46:53.749092 9124 log.cc:824] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae: Log is configured to *not* fsync() on all Append() calls
I20260620 14:46:53.750615 9124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:46:53.750855 9124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae: Bootstrap complete.
I20260620 14:46:53.752858 9124 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6e149f076a43cca9997613200965ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39179 } }
I20260620 14:46:53.753070 9124 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6e149f076a43cca9997613200965ae, State: Initialized, Role: FOLLOWER
I20260620 14:46:53.753214 9124 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6e149f076a43cca9997613200965ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39179 } }
I20260620 14:46:53.753279 9124 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:46:53.753324 9124 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:46:53.753367 9124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:46:53.754295 9124 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6e149f076a43cca9997613200965ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39179 } }
I20260620 14:46:53.754376 9124 leader_election.cc:302] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac6e149f076a43cca9997613200965ae; no voters:
I20260620 14:46:53.754522 9124 leader_election.cc:288] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:46:53.754566 9128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:46:53.754665 9128 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [term 2 LEADER]: Becoming Leader. State: Replica: ac6e149f076a43cca9997613200965ae, State: Running, Role: LEADER
I20260620 14:46:53.754751 9128 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6e149f076a43cca9997613200965ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39179 } }
I20260620 14:46:53.754773 9124 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:46:53.755016 9128 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ac6e149f076a43cca9997613200965ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6e149f076a43cca9997613200965ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39179 } } }
I20260620 14:46:53.755079 9128 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:53.755187 9128 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac6e149f076a43cca9997613200965ae. Latest consensus state: current_term: 2 leader_uuid: "ac6e149f076a43cca9997613200965ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6e149f076a43cca9997613200965ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39179 } } }
I20260620 14:46:53.755239 9128 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae [sys.catalog]: This master's current role is: LEADER
I20260620 14:46:53.755337 9132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:46:53.755837 9132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:46:53.755978 9132 catalog_manager.cc:1259] Loaded cluster ID: f21d3152f2014c32a7706bdf05a24050
I20260620 14:46:53.756188 9132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:46:53.757452 9132 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:46:53.757635 9132 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ac6e149f076a43cca9997613200965ae: Loaded TSK: 0
I20260620 14:46:53.757859 9132 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:46:53.823676 9126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:53.823864 9126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:53.823887 9126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:53.825040 9126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:53.825085 9126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:53.825111 9126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:46:53.826511 9126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33375
--tserver_master_addrs=127.0.98.62:39179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:53.826721 9126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:53.826933 9126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:53.831561 9151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:53.831676 9126 server_base.cc:1034] running on GCE node
W20260620 14:46:53.831574 9150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:53.831740 9153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:53.832041 9126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:53.832263 9126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:53.833434 9126 hybrid_clock.cc:648] HybridClock initialized: now 1781966813833417 us; error 32 us; skew 500 ppm
Jun 20 14:46:53 dist-test-slave-cx89 krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966813, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:53.835959 9126 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:46:53.836475 9126 webserver.cc:460] Webserver started at http://127.0.98.1:33375/ using document root <none> and password file <none>
I20260620 14:46:53.836735 9126 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:53.836885 9126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:53.838624 9126 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:46:53.839263 9160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:53.839448 9126 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:53.839532 9126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "6724709e381c4d33b9113b68fc7ddc5e"
format_stamp: "Formatted at 2026-06-20 14:46:53 on dist-test-slave-cx89"
server_key: "c4c829fb5be3fe9deb05e50675b93aa4"
server_key_iv: "96bb0afd853c3cdfb25b917b99da7a40"
server_key_version: "encryptionkey@0"
I20260620 14:46:53.839944 9126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:53.856837 9126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:53.859308 9126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:53.859472 9126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:53.859720 9126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:53.860061 9126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:53.860109 9126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:53.860140 9126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:53.860162 9126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:53.867144 9126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37241
I20260620 14:46:53.867178 9273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37241 every 8 connection(s)
I20260620 14:46:53.867619 9126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:46:53.868423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 9126
I20260620 14:46:53.868692 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43195
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39179
--webserver_port=36065
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:46:53 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:53 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:53.877213 9276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:53.868374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:53613)
0620 14:46:53.868551 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:53.868599 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:53.869213 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:53.870377 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:53.870382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:53.870978 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:53.871163 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:53.871168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.872784 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.872785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:53.873574 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:53.873576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.874390 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.874390 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:53.874483 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:53.874982 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:53.874992 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:53.877012 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
W20260620 14:46:53.877365 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:53.869726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53613 (local address 127.0.98.62:39179)
0620 14:46:53.869925 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:53.869935 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:53.869972 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:46:53.870096 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:53.870103 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:53.870157 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:53.870303 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:53.871277 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.872662 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.873656 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.874312 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.874433 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:53.877169 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53613: BlockingRecv error: recv got EOF from 127.0.98.1:53613 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
W20260620 14:46:53.877712 9274 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39179 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39179: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 14:46:53 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:53 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:53.881878 9276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:53.877923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:37075)
0620 14:46:53.877965 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:53.877974 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:53.878010 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:53.878139 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:53.878140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:53.878258 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:53.878304 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:53.878305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.878989 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.878990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:53.879573 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:53.879574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.879881 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.879882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:53.879941 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:53.880324 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:53.880329 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:53.881804 (+ 1475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 14:46:53.881939 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:53.877963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37075 (local address 127.0.98.62:39179)
0620 14:46:53.877994 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:53.877996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:53.878003 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:46:53.878065 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:53.878066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:53.878076 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:53.878128 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:53.878365 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.878893 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.879635 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.879821 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.879879 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:53.881888 (+ 2009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37075: BlockingRecv error: recv got EOF from 127.0.98.1:37075 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Jun 20 14:46:53 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:53 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:53.885965 9276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:53.882295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:48703)
0620 14:46:53.882382 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:53.882392 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:53.882432 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:53.882538 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:53.882539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:53.882651 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:53.882675 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:53.882676 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.883280 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.883280 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:53.883826 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:53.883827 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.884129 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.884130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:53.884175 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:53.884522 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:53.884526 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:53.885904 (+ 1378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 14:46:53.886039 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:53.882351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48703 (local address 127.0.98.62:39179)
0620 14:46:53.882383 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:53.882384 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:53.882419 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:46:53.882482 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:53.882483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:53.882493 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:53.882535 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:53.882731 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.883217 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.883891 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:53.884078 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:53.884128 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:53.885985 (+ 1857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48703: BlockingRecv error: recv got EOF from 127.0.98.1:48703 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260620 14:46:53.886240 9274 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:46:53.954252 9277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:53.954457 9277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:53.954481 9277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:53.955660 9277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:53.955700 9277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:53.955726 9277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:46:53.957266 9277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36065
--tserver_master_addrs=127.0.98.62:39179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:53.957479 9277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:53.957664 9277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:53.962395 9286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:53.962453 9277 server_base.cc:1034] running on GCE node
W20260620 14:46:53.962407 9284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:53.962409 9283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:53.962767 9277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:53.963012 9277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:53.964150 9277 hybrid_clock.cc:648] HybridClock initialized: now 1781966813964139 us; error 31 us; skew 500 ppm
Jun 20 14:46:53 dist-test-slave-cx89 krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966813, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:53.966450 9277 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:46:53.966848 9277 webserver.cc:460] Webserver started at http://127.0.98.2:36065/ using document root <none> and password file <none>
I20260620 14:46:53.967053 9277 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:53.967111 9277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:53.968420 9277 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:53.968931 9293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:53.969077 9277 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:53.969165 9277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "af4c554f688b44b194382fd537248b8e"
format_stamp: "Formatted at 2026-06-20 14:46:53 on dist-test-slave-cx89"
server_key: "709d915a27c448ac065527fd5178a006"
server_key_iv: "4ee00fe8efea683a3bde18e1830e0746"
server_key_version: "encryptionkey@0"
I20260620 14:46:53.969498 9277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:53.995219 9277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:53.997269 9277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:53.997400 9277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:53.997614 9277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:53.997895 9277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:53.997925 9277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:53.997956 9277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:53.997973 9277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:54.003177 9277 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43195
I20260620 14:46:54.003245 9406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43195 every 8 connection(s)
I20260620 14:46:54.003571 9277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:46:54.004518 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 9277
I20260620 14:46:54.004776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42355
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39179
--webserver_port=45281
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:46:54 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:54 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:46:54.010914 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:54.004295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52865 (local address 127.0.98.62:39179)
0620 14:46:54.004375 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:54.004377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:54.004762 (+ 385us) server_negotiation.cc:407] Connection header received
0620 14:46:54.005316 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:54.005317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:54.005334 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:54.005380 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:54.006277 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.006765 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.007776 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.007971 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.008025 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:54.010861 (+ 2836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52865: BlockingRecv error: recv got EOF from 127.0.98.2:52865 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:46:54.010974 9409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:54.004283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:52865)
0620 14:46:54.004635 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:54.004682 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:54.005210 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:54.005451 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:54.005455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:54.005996 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:54.006188 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:54.006192 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.006843 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.006845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:54.007702 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:54.007704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.008035 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.008037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:54.008156 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:54.008638 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:54.008646 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:54.010773 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":79,"threads_started":1}
W20260620 14:46:54.011158 9407 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39179 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39179: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 14:46:54 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:54 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:54.016142 9409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:54.011402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:39863)
0620 14:46:54.011975 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:54.011984 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:54.012028 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:54.012166 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:54.012166 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:54.012269 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:54.012314 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:54.012315 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.013159 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.013160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:54.013640 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:54.013641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.014082 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.014083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:54.014132 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:54.014512 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:54.014516 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:54.016096 (+ 1580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":558}
W20260620 14:46:54.016245 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:54.011470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39863 (local address 127.0.98.62:39179)
0620 14:46:54.011533 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:54.011534 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:54.012024 (+ 490us) server_negotiation.cc:407] Connection header received
0620 14:46:54.012091 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:54.012092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:54.012103 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:54.012152 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:54.012370 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.013080 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.013715 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.014014 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.014078 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:54.016186 (+ 2108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39863: BlockingRecv error: recv got EOF from 127.0.98.2:39863 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jun 20 14:46:54 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:54 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:54.020157 9409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:54.016462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:46729)
0620 14:46:54.016547 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:54.016556 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:54.016598 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:54.016720 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:54.016721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:54.016829 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:54.016852 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:54.016853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.017403 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.017404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:54.017878 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:54.017879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.018169 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.018170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:54.018217 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:54.018582 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:54.018586 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:54.020100 (+ 1514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 14:46:54.020272 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:54.016512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46729 (local address 127.0.98.62:39179)
0620 14:46:54.016547 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:54.016549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:54.016585 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:46:54.016659 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:54.016659 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:54.016667 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:54.016711 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:54.016910 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.017342 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.017949 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.018117 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.018165 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:54.020177 (+ 2012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46729: BlockingRecv error: recv got EOF from 127.0.98.2:46729 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 14:46:54.020270 9407 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:46:54.085042 9410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:46:54.085246 9410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:46:54.085269 9410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:46:54.086410 9410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:46:54.086457 9410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:46:54.086481 9410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:46:54.087908 9410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45281
--tserver_master_addrs=127.0.98.62:39179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:46:54.088116 9410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:46:54.088320 9410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:46:54.092914 9418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:54.092926 9416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:46:54.092922 9415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:46:54.093144 9410 server_base.cc:1034] running on GCE node
I20260620 14:46:54.093313 9410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:46:54.093492 9410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:46:54.094620 9410 hybrid_clock.cc:648] HybridClock initialized: now 1781966814094607 us; error 29 us; skew 500 ppm
Jun 20 14:46:54 dist-test-slave-cx89 krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966814, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:46:54.096868 9410 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:46:54.097249 9410 webserver.cc:460] Webserver started at http://127.0.98.3:45281/ using document root <none> and password file <none>
I20260620 14:46:54.097450 9410 fs_manager.cc:359] Metadata directory not provided
I20260620 14:46:54.097509 9410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:46:54.098748 9410 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:46:54.099283 9425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:54.099455 9410 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:46:54.099532 9410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "273c591cfcbb4fcd83a2be8a97958326"
format_stamp: "Formatted at 2026-06-20 14:46:53 on dist-test-slave-cx89"
server_key: "03e3ab58f7bcddcf321033c54e9a0122"
server_key_iv: "93e1255588800b5a3bce014cbce3f2ac"
server_key_version: "encryptionkey@0"
I20260620 14:46:54.099838 9410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:46:54.119190 9410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:46:54.121411 9410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:46:54.121567 9410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:46:54.121786 9410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:46:54.122184 9410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:46:54.122229 9410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:54.122259 9410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:46:54.122277 9410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:46:54.127507 9410 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42355
I20260620 14:46:54.127585 9538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42355 every 8 connection(s)
I20260620 14:46:54.127878 9410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:46:54.129202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 9410
Jun 20 14:46:54 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:54 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:46:54.134881 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:54.128522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47877 (local address 127.0.98.62:39179)
0620 14:46:54.128602 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:54.128604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:54.128765 (+ 161us) server_negotiation.cc:407] Connection header received
0620 14:46:54.129322 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:54.129323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:54.129334 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:54.129388 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:54.130427 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.130903 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.131902 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.132080 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.132131 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:54.134835 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47877: BlockingRecv error: recv got EOF from 127.0.98.3:47877 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:46:54.134986 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:54.128505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.3:47877)
0620 14:46:54.128647 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:54.128682 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:54.129207 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:54.129460 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:54.129464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:54.130152 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:54.130345 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:54.130349 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.130977 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.130978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:54.131832 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:54.131834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.132143 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.132144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:54.132259 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:54.132585 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:54.132594 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:54.134772 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
W20260620 14:46:54.135183 9539 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39179 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39179: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260620 14:46:54.136399 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:54.129719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.0.98.62:39179)
0620 14:46:54.129826 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:54.129828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:54.129836 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:46:54.129858 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:54.129859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:54.129870 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:54.129942 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:54.130470 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.130950 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.131703 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.131904 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.131957 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:54.133128 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:46:54.133133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:46:54.133133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:46:54.133144 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:46:54.134976 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:54.135454 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:54.135457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:54.135457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:54.135576 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:46:54.135767 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:46:54.135768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:46:54.135768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:46:54.136025 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:46:54.136119 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:46:54.136245 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 14:46:54.136325 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
Jun 20 14:46:54 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:54 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:54.139488 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:54.135391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.3:51621)
0620 14:46:54.135457 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:54.135468 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:54.135502 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:54.135601 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:54.135601 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:54.135722 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:54.135771 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:54.135773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.136694 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.136695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:54.137268 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:54.137270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.137572 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.137572 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:54.137634 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:54.137948 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:54.137952 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:54.139429 (+ 1477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 14:46:54.139559 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:54.135459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51621 (local address 127.0.98.62:39179)
0620 14:46:54.135501 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:54.135503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:54.135511 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:46:54.135548 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:54.135548 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:54.135555 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:54.135596 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:54.135833 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.136409 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.137332 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.137519 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.137568 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:54.139511 (+ 1943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51621: BlockingRecv error: recv got EOF from 127.0.98.3:51621 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 14:46:54 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:54 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:54.143134 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:54.139694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.3:59343)
0620 14:46:54.139800 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:54.139809 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:54.139857 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:54.139970 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:54.139970 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:54.140077 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:54.140102 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:54.140103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.140651 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.140651 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:54.141137 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:54.141138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.141431 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.141431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:54.141480 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:54.141771 (+ 291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:54.141774 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:54.143075 (+ 1301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
W20260620 14:46:54.143217 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:54.139734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59343 (local address 127.0.98.62:39179)
0620 14:46:54.139786 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:54.139788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:54.139847 (+ 59us) server_negotiation.cc:407] Connection header received
0620 14:46:54.139907 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:54.139908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:54.139917 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:54.139966 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:54.140158 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.140591 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.141200 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.141379 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.141427 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:54.143171 (+ 1744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59343: BlockingRecv error: recv got EOF from 127.0.98.3:59343 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 14:46:54.143241 9539 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 14:46:54 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:54 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:54.891376 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:54.886648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:35819)
0620 14:46:54.886774 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:54.886784 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:54.886834 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:54.887043 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:54.887045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:54.887209 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:54.887262 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:54.887264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.888019 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.888020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:54.888576 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:54.888579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.888972 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.888973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:54.889024 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:54.889501 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:54.889506 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:54.891299 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
W20260620 14:46:54.891526 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:54.886753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35819 (local address 127.0.98.62:39179)
0620 14:46:54.886888 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:54.886890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:54.886898 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:46:54.886923 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:54.886924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:54.886934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:54.886998 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:54.887357 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.887915 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.888666 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:54.888876 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:54.888939 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:54.891453 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35819: BlockingRecv error: recv got EOF from 127.0.98.1:35819 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
Jun 20 14:46:55 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:55 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:55.025138 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:55.020647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:54967)
0620 14:46:55.020812 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:55.020827 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:55.020914 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:55.021086 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:55.021088 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:55.021268 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:55.021318 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:55.021320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:55.021968 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:55.021969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:55.022513 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:55.022515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:55.022864 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:55.022865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:55.022921 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:55.023399 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:55.023404 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:55.025050 (+ 1646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":60,"threads_started":1}
W20260620 14:46:55.025195 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:55.020727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54967 (local address 127.0.98.62:39179)
0620 14:46:55.020817 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:55.020819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:55.020904 (+ 85us) server_negotiation.cc:407] Connection header received
0620 14:46:55.021004 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:55.021005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:55.021019 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:55.021075 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:55.021394 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:55.021885 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:55.022600 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:55.022797 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:55.022860 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:55.025143 (+ 2283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54967: BlockingRecv error: recv got EOF from 127.0.98.2:54967 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 14:46:55 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:55 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:55.148046 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:55.143645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.3:46725)
0620 14:46:55.143757 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:55.143782 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:55.143850 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:55.144019 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:55.144020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:55.144195 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:55.144242 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:55.144244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:55.144929 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:55.144931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:55.145512 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:55.145514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:55.145878 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:55.145878 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:55.145923 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:55.146357 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:55.146361 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:55.147981 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":61,"threads_started":1}
W20260620 14:46:55.148159 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:55.143683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46725 (local address 127.0.98.62:39179)
0620 14:46:55.143743 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:55.143746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:55.143848 (+ 102us) server_negotiation.cc:407] Connection header received
0620 14:46:55.143941 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:55.143942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:55.143955 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:55.144005 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:55.144302 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:55.144831 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:55.145607 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:55.145807 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:55.145870 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:55.148090 (+ 2220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46725: BlockingRecv error: recv got EOF from 127.0.98.3:46725 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 14:46:55 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:55 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:55.895967 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:55.891803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:34903)
0620 14:46:55.891910 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:55.891921 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:55.891969 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:55.892112 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:55.892113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:55.892251 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:55.892297 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:55.892299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:55.892921 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:55.892922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:55.893447 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:55.893448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:55.893770 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:55.893771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:55.893835 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:55.894247 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:55.894251 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:55.895902 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":54,"threads_started":1}
W20260620 14:46:55.896102 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:55.891898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34903 (local address 127.0.98.62:39179)
0620 14:46:55.892007 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:55.892009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:55.892016 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:46:55.892035 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:55.892036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:55.892045 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:55.892104 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:55.892354 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:55.892846 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:55.893510 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:55.893691 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:55.893737 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:55.896031 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34903: BlockingRecv error: recv got EOF from 127.0.98.1:34903 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
Jun 20 14:46:56 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:56 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:56.030117 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:56.025646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:53779)
0620 14:46:56.025806 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:56.025820 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:56.025905 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:56.026047 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:56.026048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:56.026228 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:56.026293 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:56.026295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:56.026957 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:56.026958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:56.027535 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:56.027538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:56.027897 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:56.027899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:56.027953 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:56.028374 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:56.028378 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:56.030050 (+ 1672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":69,"threads_started":1}
W20260620 14:46:56.030201 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:56.025788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53779 (local address 127.0.98.62:39179)
0620 14:46:56.025849 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:56.025850 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:56.025882 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:46:56.025967 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:56.025968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:56.025982 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:56.026036 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:56.026372 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:56.026878 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:56.027634 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:56.027830 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:56.027891 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:56.030139 (+ 2248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53779: BlockingRecv error: recv got EOF from 127.0.98.2:53779 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 14:46:56 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:56 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:56.152809 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:56.148511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.3:45655)
0620 14:46:56.148635 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:56.148645 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:56.148690 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:56.148822 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:56.148822 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:56.148969 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:56.149013 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:56.149015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:56.149632 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:56.149632 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:56.150140 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:56.150142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:56.150500 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:56.150501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:56.150550 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:56.151038 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:56.151043 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:56.152752 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
W20260620 14:46:56.152931 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:56.148657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45655 (local address 127.0.98.62:39179)
0620 14:46:56.148710 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:56.148712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:56.148718 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:46:56.148739 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:56.148739 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:56.148750 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:56.148812 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:56.149084 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:56.149576 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:56.150211 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:56.150410 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:56.150463 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:56.152856 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45655: BlockingRecv error: recv got EOF from 127.0.98.3:45655 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 14:46:56 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:56 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:56.900563 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:56.896418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:51403)
0620 14:46:56.896543 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:56.896552 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:56.896600 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:56.896741 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:56.896742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:56.896886 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:56.896935 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:56.896937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:56.897565 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:56.897565 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:56.898082 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:56.898083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:56.898484 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:56.898485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:56.898529 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:56.898911 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:56.898915 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:56.900474 (+ 1559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
W20260620 14:46:56.900619 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:56.896513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51403 (local address 127.0.98.62:39179)
0620 14:46:56.896628 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:56.896630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:56.896638 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:46:56.896659 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:56.896660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:56.896669 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:56.896731 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:56.896988 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:56.897478 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:56.898134 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:56.898418 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:56.898480 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:56.900557 (+ 2077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51403: BlockingRecv error: recv got EOF from 127.0.98.1:51403 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
Jun 20 14:46:57 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:57 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:57.034986 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:57.030582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:53989)
0620 14:46:57.030733 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:57.030745 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:57.030807 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:57.030921 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:57.030922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:57.031081 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:57.031132 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:57.031133 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:57.031810 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:57.031811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:57.032334 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:57.032336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:57.032700 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:57.032702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:57.032749 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:57.033233 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:57.033237 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:57.034909 (+ 1672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
W20260620 14:46:57.035050 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:57.030685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53989 (local address 127.0.98.62:39179)
0620 14:46:57.030758 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:57.030760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:57.030789 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:46:57.030854 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:57.030855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:57.030865 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:57.030916 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:57.031182 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:57.031742 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:57.032420 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:57.032625 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:57.032681 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:57.034999 (+ 2318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53989: BlockingRecv error: recv got EOF from 127.0.98.2:53989 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 14:46:57 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:57 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:57.157884 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:57.153269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.3:38421)
0620 14:46:57.153388 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:57.153400 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:57.153470 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:57.153622 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:57.153623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:57.153815 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:57.153871 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:57.153874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:57.154664 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:57.154665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:57.155301 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:57.155304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:57.155751 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:57.155751 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:57.155793 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:57.156199 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:57.156204 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:57.157813 (+ 1609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
W20260620 14:46:57.157953 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:57.153344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38421 (local address 127.0.98.62:39179)
0620 14:46:57.153392 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:57.153394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:57.153452 (+ 58us) server_negotiation.cc:407] Connection header received
0620 14:46:57.153545 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:57.153546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:57.153560 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:57.153611 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:57.153952 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:57.154588 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:57.155380 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:57.155681 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:57.155748 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:57.157890 (+ 2142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38421: BlockingRecv error: recv got EOF from 127.0.98.3:38421 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 14:46:57 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:57 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:57.905617 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:57.901020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:33041)
0620 14:46:57.901171 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:57.901182 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:57.901236 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:57.901347 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:57.901348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:57.901511 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:57.901558 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:57.901559 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:57.902234 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:57.902236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:57.902877 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:57.902881 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:57.903256 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:57.903257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:57.903295 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:57.903918 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:57.903922 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:57.905540 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
W20260620 14:46:57.905696 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:57.901122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33041 (local address 127.0.98.62:39179)
0620 14:46:57.901225 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:57.901227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:57.901235 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:46:57.901281 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:57.901282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:57.901292 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:57.901351 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:57.901638 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:57.902130 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:57.902988 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:57.903190 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:57.903255 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:57.905633 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33041: BlockingRecv error: recv got EOF from 127.0.98.1:33041 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":48,"threads_started":1}
Jun 20 14:46:58 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:58 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:58.042372 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:58.035523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:46969)
0620 14:46:58.035630 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:58.035643 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:58.035719 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:58.035856 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:58.035857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:58.036016 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:58.036060 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:58.036062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:58.036828 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:58.036829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:58.037356 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:58.037358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:58.037682 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:58.037683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:58.037735 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:58.040674 (+ 2939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:58.040678 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:58.042294 (+ 1616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":57,"threads_started":1}
W20260620 14:46:58.042445 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:58.035595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46969 (local address 127.0.98.62:39179)
0620 14:46:58.035652 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:58.035654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:58.035699 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:46:58.035780 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:58.035781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:58.035793 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:58.035851 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:58.036140 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:58.036765 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:58.037430 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:58.037626 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:58.037679 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:58.042392 (+ 4713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46969: BlockingRecv error: recv got EOF from 127.0.98.2:46969 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 14:46:58 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:58 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:58.162487 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:58.158295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.3:58781)
0620 14:46:58.158420 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:58.158432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:58.158486 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:58.158590 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:58.158590 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:58.158737 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:58.158783 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:58.158785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:58.159379 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:58.159380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:58.159930 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:58.159932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:58.160250 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:58.160250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:58.160286 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:58.160697 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:58.160701 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:58.162427 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":81,"threads_started":1}
W20260620 14:46:58.162577 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:58.158360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58781 (local address 127.0.98.62:39179)
0620 14:46:58.158428 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:58.158430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:58.158481 (+ 51us) server_negotiation.cc:407] Connection header received
0620 14:46:58.158524 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:58.158525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:58.158537 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:58.158585 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:58.158826 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:58.159308 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:58.160012 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:58.160196 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:58.160248 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:58.162512 (+ 2264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58781: BlockingRecv error: recv got EOF from 127.0.98.3:58781 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 14:46:58 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:58 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:58.910480 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:58.906108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:33813)
0620 14:46:58.906229 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:58.906239 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:58.906295 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:58.906434 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:58.906434 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:58.906600 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:58.906651 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:58.906653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:58.907268 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:58.907269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:58.907856 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:58.907858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:58.908199 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:58.908201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:58.908265 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:58.908709 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:58.908714 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:58.910394 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
W20260620 14:46:58.910569 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:58.906192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33813 (local address 127.0.98.62:39179)
0620 14:46:58.906293 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:58.906294 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:58.906303 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:46:58.906330 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:58.906331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:58.906340 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:58.906419 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:58.906715 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:58.907209 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:58.907923 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:58.908115 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:58.908176 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:58.910510 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33813: BlockingRecv error: recv got EOF from 127.0.98.1:33813 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":47,"threads_started":1}
Jun 20 14:46:59 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:59 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:59.047248 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:59.042863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:55797)
0620 14:46:59.043030 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:59.043044 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:59.043110 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:59.043215 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:59.043216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:59.043368 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:59.043449 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:59.043450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:59.044069 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:59.044070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:59.044608 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:59.044610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:59.044921 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:59.044924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:59.044989 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:59.045455 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:59.045459 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:59.047168 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":53,"threads_started":1}
W20260620 14:46:59.047379 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:59.043013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55797 (local address 127.0.98.62:39179)
0620 14:46:59.043064 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:59.043066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:59.043093 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:46:59.043153 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:59.043154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:59.043164 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:59.043209 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:59.043490 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:59.044009 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:59.044666 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:59.044845 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:59.044892 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:59.047260 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55797: BlockingRecv error: recv got EOF from 127.0.98.2:55797 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 14:46:59 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:59 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:59.167529 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:59.162929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.3:48227)
0620 14:46:59.163084 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:59.163095 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:59.163168 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:59.163289 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:59.163290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:59.163486 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:59.163540 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:59.163542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:59.164211 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:59.164212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:59.164761 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:59.164764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:59.165166 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:59.165168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:59.165228 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:59.165695 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:59.165699 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:59.167473 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
W20260620 14:46:59.167625 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:59.163037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48227 (local address 127.0.98.62:39179)
0620 14:46:59.163093 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:59.163095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:59.163161 (+ 66us) server_negotiation.cc:407] Connection header received
0620 14:46:59.163214 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:59.163215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:59.163226 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:59.163272 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:59.163599 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:59.164127 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:59.164858 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:59.165095 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:59.165161 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:59.167558 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48227: BlockingRecv error: recv got EOF from 127.0.98.3:48227 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 14:46:59 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:46:59 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:46:59.915246 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:46:59.910951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:50789)
0620 14:46:59.911061 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 14:46:59.911071 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:46:59.911116 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:46:59.911288 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:46:59.911289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:46:59.911477 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:46:59.911527 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:59.911529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:59.912190 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:59.912191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:59.912730 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:46:59.912732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:46:59.913058 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:46:59.913058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:46:59.913098 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:59.913526 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:46:59.913530 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:46:59.915163 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":54,"threads_started":1}
W20260620 14:46:59.915361 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:46:59.911052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50789 (local address 127.0.98.62:39179)
0620 14:46:59.911170 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:46:59.911171 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:46:59.911179 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:46:59.911201 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:46:59.911202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:46:59.911212 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:46:59.911274 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:46:59.911600 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:59.912117 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:59.912801 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:46:59.912997 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:46:59.913055 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:46:59.915294 (+ 2239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50789: BlockingRecv error: recv got EOF from 127.0.98.1:50789 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
Jun 20 14:47:00 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:00 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:00.052006 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:00.047705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:47425)
0620 14:47:00.047832 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:00.047842 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:00.047904 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:00.048022 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:00.048022 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:00.048171 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:00.048219 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:00.048221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:00.048834 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:00.048835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:00.049420 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:00.049422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:00.049787 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:00.049788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:00.049839 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:00.050294 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:00.050298 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:00.051947 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
W20260620 14:47:00.052124 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:00.047791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47425 (local address 127.0.98.62:39179)
0620 14:47:00.047835 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:00.047836 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:00.047885 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:47:00.047959 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:00.047960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:00.047970 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:00.048008 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:00.048270 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:00.048748 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:00.049512 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:00.049703 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:00.049757 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:00.052063 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47425: BlockingRecv error: recv got EOF from 127.0.98.2:47425 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 14:47:00 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:00 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:00.172246 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:00.167965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.3:58645)
0620 14:47:00.168123 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:00.168137 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:00.168209 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:00.168336 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:00.168337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:00.168504 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:00.168547 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:00.168549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:00.169183 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:00.169184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:00.169712 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:00.169714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:00.170021 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:00.170022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:00.170071 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:00.170488 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:00.170492 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:00.172178 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":74,"threads_started":1}
W20260620 14:47:00.172315 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:00.168051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58645 (local address 127.0.98.62:39179)
0620 14:47:00.168133 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:00.168135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:00.168197 (+ 62us) server_negotiation.cc:407] Connection header received
0620 14:47:00.168261 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:00.168262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:00.168276 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:00.168316 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:00.168613 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:00.169114 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:00.169783 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:00.169965 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:00.170018 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:00.172257 (+ 2239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58645: BlockingRecv error: recv got EOF from 127.0.98.3:58645 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 14:47:00 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:00 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:00.920223 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:00.915775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:56691)
0620 14:47:00.915904 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:00.915918 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:00.915983 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:00.916114 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:00.916115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:00.916274 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:00.916324 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:00.916326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:00.916974 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:00.916975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:00.917524 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:00.917526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:00.917869 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:00.917869 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:00.917905 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:00.918332 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:00.918335 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:00.920140 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
W20260620 14:47:00.920302 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:00.915846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56691 (local address 127.0.98.62:39179)
0620 14:47:00.915984 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:00.915985 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:00.915994 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:47:00.916037 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:00.916038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:00.916049 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:00.916106 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:00.916403 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:00.916910 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:00.917606 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:00.917806 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:00.917873 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:00.920231 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56691: BlockingRecv error: recv got EOF from 127.0.98.1:56691 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
Jun 20 14:47:01 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:01 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:01.056731 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:01.052445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:38989)
0620 14:47:01.052587 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:01.052599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:01.052674 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:01.052798 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:01.052799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:01.052963 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:01.053014 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:01.053015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:01.053601 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:01.053601 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:01.054123 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:01.054125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:01.054396 (+ 271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:01.054397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:01.054444 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:01.054868 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:01.054872 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:01.056670 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
W20260620 14:47:01.056834 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:01.052541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38989 (local address 127.0.98.62:39179)
0620 14:47:01.052587 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:01.052589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:01.052665 (+ 76us) server_negotiation.cc:407] Connection header received
0620 14:47:01.052729 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:01.052729 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:01.052739 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:01.052785 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:01.053068 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:01.053548 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:01.054180 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:01.054349 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:01.054390 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:01.056772 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38989: BlockingRecv error: recv got EOF from 127.0.98.2:38989 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 14:47:01 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:01 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:01.176911 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:01.172667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.3:43083)
0620 14:47:01.172801 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:01.172812 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:01.172878 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:01.173014 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:01.173015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:01.173195 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:01.173242 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:01.173244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:01.173883 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:01.173884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:01.174410 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:01.174412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:01.174733 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:01.174733 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:01.174786 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:01.175205 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:01.175210 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:01.176836 (+ 1626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
W20260620 14:47:01.176985 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:01.172718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43083 (local address 127.0.98.62:39179)
0620 14:47:01.172771 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:01.172773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:01.172866 (+ 93us) server_negotiation.cc:407] Connection header received
0620 14:47:01.172938 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:01.172939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:01.172953 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:01.173002 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:01.173307 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:01.173812 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:01.174489 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:01.174677 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:01.174731 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:01.176928 (+ 2197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43083: BlockingRecv error: recv got EOF from 127.0.98.3:43083 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 20 14:47:01 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:01 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:01.925179 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:01.920640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:49401)
0620 14:47:01.920790 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:01.920801 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:01.920865 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:01.921016 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:01.921017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:01.921166 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:01.921219 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:01.921220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:01.922084 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:01.922086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:01.922619 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:01.922621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:01.922933 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:01.922933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:01.922969 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:01.923435 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:01.923440 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:01.925095 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
W20260620 14:47:01.925257 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:01.920750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49401 (local address 127.0.98.62:39179)
0620 14:47:01.920891 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:01.920893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:01.920902 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:47:01.920926 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:01.920927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:01.920938 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:01.921007 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:01.921305 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:01.922005 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:01.922689 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:01.922882 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:01.922931 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:01.925193 (+ 2262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49401: BlockingRecv error: recv got EOF from 127.0.98.1:49401 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
Jun 20 14:47:02 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:02 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:02.061529 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:02.057180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:43775)
0620 14:47:02.057320 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:02.057331 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:02.057392 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:02.057508 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:02.057509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:02.057655 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:02.057701 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:02.057702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:02.058425 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:02.058426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:02.058949 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:02.058951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:02.059317 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:02.059319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:02.059381 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:02.059814 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:02.059819 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:02.061456 (+ 1637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
W20260620 14:47:02.061632 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:02.057234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43775 (local address 127.0.98.62:39179)
0620 14:47:02.057287 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:02.057289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:02.057384 (+ 95us) server_negotiation.cc:407] Connection header received
0620 14:47:02.057432 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:02.057434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:02.057445 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:02.057491 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:02.057747 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:02.058366 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:02.059030 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:02.059231 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:02.059291 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:02.061572 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43775: BlockingRecv error: recv got EOF from 127.0.98.2:43775 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 20 14:47:02 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:02 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:02.181649 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:02.177354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.3:59655)
0620 14:47:02.177475 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:02.177488 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:02.177560 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:02.177693 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:02.177694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:02.177862 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:02.177907 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:02.177909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:02.178522 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:02.178523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:02.179029 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:02.179031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:02.179335 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:02.179336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:02.179388 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:02.179836 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:02.179841 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:02.181593 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
W20260620 14:47:02.181758 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:02.177444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59655 (local address 127.0.98.62:39179)
0620 14:47:02.177500 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:02.177502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:02.177541 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:47:02.177627 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:02.177628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:02.177639 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:02.177682 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:02.177976 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:02.178454 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:02.179100 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:02.179278 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:02.179330 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:02.181688 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59655: BlockingRecv error: recv got EOF from 127.0.98.3:59655 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 14:47:02 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:02 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:02.930336 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:02.925674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:51799)
0620 14:47:02.925828 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:02.925839 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:02.925894 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:02.926064 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:02.926065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:02.926233 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:02.926298 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:02.926300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:02.926992 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:02.926993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:02.927566 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:02.927568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:02.927984 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:02.927985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:02.928055 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:02.928572 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:02.928576 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:02.930271 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
W20260620 14:47:02.930455 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:02.925772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51799 (local address 127.0.98.62:39179)
0620 14:47:02.925876 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:02.925878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:02.925887 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:47:02.925950 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:02.925951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:02.925961 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:02.926018 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:02.926400 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:02.926914 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:02.927652 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:02.927903 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:02.927985 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:02.930379 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51799: BlockingRecv error: recv got EOF from 127.0.98.1:51799 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":47,"threads_started":1}
Jun 20 14:47:03 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:03 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:03.070314 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:03.061962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:50691)
0620 14:47:03.062103 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:03.062124 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:03.062196 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:03.062323 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:03.062324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:03.062493 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:03.062541 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:03.062543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:03.063169 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:03.063171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:03.063718 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:03.063720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:03.064041 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:03.064042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:03.064095 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:03.068457 (+ 4362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:03.068462 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:03.070254 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
W20260620 14:47:03.070417 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:03.062048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50691 (local address 127.0.98.62:39179)
0620 14:47:03.062131 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:03.062132 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:03.062177 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:47:03.062246 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:03.062247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:03.062259 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:03.062307 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:03.062613 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:03.063102 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:03.063793 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:03.063981 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:03.064033 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:03.070357 (+ 6324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50691: BlockingRecv error: recv got EOF from 127.0.98.2:50691 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 14:47:03 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:03 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:03.186488 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:03.182079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.3:38957)
0620 14:47:03.182201 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:03.182217 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:03.182287 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:03.182407 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:03.182408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:03.182576 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:03.182622 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:03.182624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:03.183243 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:03.183244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:03.183810 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:03.183812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:03.184140 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:03.184141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:03.184195 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:03.184619 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:03.184624 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:03.186414 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
W20260620 14:47:03.186555 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:03.182144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38957 (local address 127.0.98.62:39179)
0620 14:47:03.182213 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:03.182215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:03.182275 (+ 60us) server_negotiation.cc:407] Connection header received
0620 14:47:03.182339 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:03.182340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:03.182354 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:03.182401 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:03.182686 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:03.183177 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:03.183884 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:03.184081 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:03.184136 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:03.186502 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38957: BlockingRecv error: recv got EOF from 127.0.98.3:38957 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 14:47:03 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:03 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:03.935053 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:03.930805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:36587)
0620 14:47:03.930934 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:03.930943 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:03.930992 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:03.931130 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:03.931131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:03.931340 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:03.931399 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:03.931401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:03.932072 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:03.932073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:03.932603 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:03.932605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:03.932893 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:03.932894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:03.932933 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:03.933361 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:03.933366 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:03.934983 (+ 1617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
W20260620 14:47:03.935150 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:03.930863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36587 (local address 127.0.98.62:39179)
0620 14:47:03.930992 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:03.930994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:03.931002 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:47:03.931044 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:03.931045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:03.931054 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:03.931108 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:03.931483 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:03.932010 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:03.932659 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:03.932843 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:03.932885 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:03.935075 (+ 2190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36587: BlockingRecv error: recv got EOF from 127.0.98.1:36587 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":72,"threads_started":1}
Jun 20 14:47:04 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:04 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:04.075070 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:04.070762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:60179)
0620 14:47:04.070901 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:04.070911 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:04.070965 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:04.071074 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:04.071075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:04.071234 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:04.071282 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:04.071283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:04.071998 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:04.071999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:04.072516 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:04.072517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:04.072844 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:04.072844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:04.072881 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:04.073315 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:04.073320 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:04.074991 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
W20260620 14:47:04.075153 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:04.070819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60179 (local address 127.0.98.62:39179)
0620 14:47:04.070869 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:04.070871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:04.070960 (+ 89us) server_negotiation.cc:407] Connection header received
0620 14:47:04.071006 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:04.071006 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:04.071016 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:04.071061 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:04.071362 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:04.071941 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:04.072584 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:04.072780 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:04.072846 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:04.075101 (+ 2255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60179: BlockingRecv error: recv got EOF from 127.0.98.2:60179 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 14:47:04 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:04 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:04.191005 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:04.186965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.3:50967)
0620 14:47:04.187094 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:04.187104 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:04.187160 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:04.187273 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:04.187273 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:04.187442 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:04.187489 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:04.187491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:04.188073 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:04.188074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:04.188601 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:04.188602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:04.188906 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:04.188906 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:04.188942 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:04.189350 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:04.189355 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:04.190949 (+ 1594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
W20260620 14:47:04.191102 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:04.187056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50967 (local address 127.0.98.62:39179)
0620 14:47:04.187105 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:04.187106 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:04.187150 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:47:04.187206 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:04.187207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:04.187216 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:04.187261 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:04.187541 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:04.188019 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:04.188672 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:04.188855 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:04.188904 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:04.191041 (+ 2137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50967: BlockingRecv error: recv got EOF from 127.0.98.3:50967 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 20 14:47:04 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:04 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:04.939756 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:04.935575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:60701)
0620 14:47:04.935683 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:04.935694 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:04.935751 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:04.935886 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:04.935886 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:04.936067 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:04.936125 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:04.936128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:04.936776 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:04.936777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:04.937301 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:04.937302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:04.937579 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:04.937579 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:04.937620 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:04.938009 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:04.938013 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:04.939677 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":59,"threads_started":1}
W20260620 14:47:04.939831 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:04.935660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60701 (local address 127.0.98.62:39179)
0620 14:47:04.935766 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:04.935768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:04.935776 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:47:04.935800 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:04.935801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:04.935813 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:04.935883 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:04.936194 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:04.936710 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:04.937360 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:04.937533 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:04.937576 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:04.939769 (+ 2193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60701: BlockingRecv error: recv got EOF from 127.0.98.1:60701 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":55,"threads_started":1}
Jun 20 14:47:05 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:05 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:05.079730 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:05.075556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:44709)
0620 14:47:05.075704 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:05.075714 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:05.075776 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:05.075898 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:05.075899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:05.076067 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:05.076112 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:05.076114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:05.076767 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:05.076769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:05.077286 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:05.077287 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:05.077607 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:05.077609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:05.077662 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:05.078065 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:05.078069 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:05.079653 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
W20260620 14:47:05.079794 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:05.075593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44709 (local address 127.0.98.62:39179)
0620 14:47:05.075661 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:05.075664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:05.075770 (+ 106us) server_negotiation.cc:407] Connection header received
0620 14:47:05.075829 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:05.075830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:05.075841 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:05.075883 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:05.076189 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:05.076697 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:05.077356 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:05.077552 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:05.077603 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:05.079742 (+ 2139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44709: BlockingRecv error: recv got EOF from 127.0.98.2:44709 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 14:47:05 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:05 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:05.195839 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:05.191504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.3:41129)
0620 14:47:05.191628 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:05.191648 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:05.191716 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:05.191850 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:05.191851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:05.192010 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:05.192055 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:05.192057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:05.192699 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:05.192700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:05.193224 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:05.193226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:05.193599 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:05.193601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:05.193674 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:05.194122 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:05.194127 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:05.195765 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
W20260620 14:47:05.195907 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:05.191615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41129 (local address 127.0.98.62:39179)
0620 14:47:05.191703 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:05.191705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:05.191715 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:47:05.191780 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:05.191781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:05.191794 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:05.191836 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:05.192127 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:05.192627 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:05.193300 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:05.193489 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:05.193553 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:05.195853 (+ 2300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41129: BlockingRecv error: recv got EOF from 127.0.98.3:41129 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 14:47:05 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:05 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:05.944650 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:05.940247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:42957)
0620 14:47:05.940378 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:05.940389 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:05.940447 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:05.940566 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:05.940567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:05.940723 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:05.940771 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:05.940773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:05.941420 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:05.941421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:05.941967 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:05.941969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:05.942315 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:05.942316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:05.942361 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:05.942840 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:05.942845 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:05.944553 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
W20260620 14:47:05.944694 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:05.940342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42957 (local address 127.0.98.62:39179)
0620 14:47:05.940442 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:05.940443 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:05.940453 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:47:05.940493 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:05.940494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:05.940503 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:05.940556 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:05.940828 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:05.941333 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:05.942047 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:05.942247 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:05.942299 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:05.944641 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42957: BlockingRecv error: recv got EOF from 127.0.98.1:42957 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":46,"threads_started":1}
Jun 20 14:47:06 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:06 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:06.084473 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:06.080249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:50621)
0620 14:47:06.080409 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:06.080421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:06.080484 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:06.080602 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:06.080602 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:06.080768 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:06.080814 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:06.080816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:06.081478 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:06.081478 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:06.082007 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:06.082009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:06.082308 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:06.082309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:06.082352 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:06.082752 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:06.082756 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:06.084384 (+ 1628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
W20260620 14:47:06.084534 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:06.080333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50621 (local address 127.0.98.62:39179)
0620 14:47:06.080407 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:06.080409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:06.080478 (+ 69us) server_negotiation.cc:407] Connection header received
0620 14:47:06.080539 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:06.080540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:06.080549 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:06.080589 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:06.080905 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:06.081404 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:06.082068 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:06.082249 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:06.082292 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:06.084483 (+ 2191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50621: BlockingRecv error: recv got EOF from 127.0.98.2:50621 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 14:47:06 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:06 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:06.200733 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:06.196351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.3:59255)
0620 14:47:06.196459 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:06.196469 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:06.196531 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:06.196643 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:06.196644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:06.196794 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:06.196842 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:06.196844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:06.197457 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:06.197458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:06.197990 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:06.197992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:06.198289 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:06.198291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:06.198342 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:06.198743 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:06.198747 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:06.200659 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":62,"threads_started":1}
W20260620 14:47:06.200798 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:06.196438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59255 (local address 127.0.98.62:39179)
0620 14:47:06.196481 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:06.196483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:06.196519 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:47:06.196572 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:06.196573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:06.196586 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:06.196629 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:06.196914 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:06.197393 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:06.198047 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:06.198212 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:06.198256 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:06.200742 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59255: BlockingRecv error: recv got EOF from 127.0.98.3:59255 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 14:47:06 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:06 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:06.949617 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:06.945012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:52783)
0620 14:47:06.945155 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:06.945167 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:06.945221 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:06.945465 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:06.945466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:06.945650 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:06.945706 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:06.945708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:06.946389 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:06.946390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:06.946947 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:06.946949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:06.947337 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:06.947339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:06.947391 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:06.947873 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:06.947879 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:06.949525 (+ 1646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
W20260620 14:47:06.949716 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:06.945062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52783 (local address 127.0.98.62:39179)
0620 14:47:06.945170 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:06.945172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:06.945211 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:47:06.945324 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:06.945325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:06.945337 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:06.945407 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:06.945806 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:06.946306 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:06.947034 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:06.947243 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:06.947308 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:06.949649 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52783: BlockingRecv error: recv got EOF from 127.0.98.1:52783 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
Jun 20 14:47:07 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:07 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:07.088908 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:07.084910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:33213)
0620 14:47:07.085046 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:07.085059 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:07.085116 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:07.085218 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:07.085218 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:07.085374 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:07.085421 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:07.085422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:07.086002 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:07.086003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:07.086519 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:07.086520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:07.086790 (+ 270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:07.086791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:07.086825 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:07.087265 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:07.087270 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:07.088851 (+ 1581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
W20260620 14:47:07.088999 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:07.085024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33213 (local address 127.0.98.62:39179)
0620 14:47:07.085065 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:07.085067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:07.085101 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:47:07.085167 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:07.085168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:07.085177 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:07.085212 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:07.085469 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:07.085950 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:07.086570 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:07.086744 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:07.086785 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:07.088939 (+ 2154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33213: BlockingRecv error: recv got EOF from 127.0.98.2:33213 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 14:47:07 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:07 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:07.205387 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:07.201217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.3:56109)
0620 14:47:07.201345 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:07.201355 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:07.201405 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:07.201524 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:07.201524 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:07.201673 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:07.201718 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:07.201719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:07.202334 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:07.202335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:07.202885 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:07.202888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:07.203210 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:07.203210 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:07.203245 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:07.203709 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:07.203714 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:07.205328 (+ 1614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
W20260620 14:47:07.205502 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:07.201271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56109 (local address 127.0.98.62:39179)
0620 14:47:07.201322 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:07.201323 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:07.201402 (+ 79us) server_negotiation.cc:407] Connection header received
0620 14:47:07.201445 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:07.201446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:07.201459 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:07.201516 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:07.201771 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:07.202270 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:07.202952 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:07.203149 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:07.203202 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:07.205437 (+ 2235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56109: BlockingRecv error: recv got EOF from 127.0.98.3:56109 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 14:47:07 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:07 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:07.954377 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:07.950139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:45823)
0620 14:47:07.950252 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:07.950264 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:07.950326 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:07.950467 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:07.950468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:07.950644 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:07.950700 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:07.950702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:07.951350 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:07.951352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:07.951890 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:07.951892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:07.952217 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:07.952218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:07.952267 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:07.952695 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:07.952700 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:07.954306 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
W20260620 14:47:07.954506 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:07.950216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45823 (local address 127.0.98.62:39179)
0620 14:47:07.950340 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:07.950342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:07.950350 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:47:07.950376 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:07.950377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:07.950389 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:07.950457 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:07.950764 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:07.951264 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:07.951963 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:07.952159 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:07.952213 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:07.954430 (+ 2217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45823: BlockingRecv error: recv got EOF from 127.0.98.1:45823 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
Jun 20 14:47:08 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:08 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:08.097311 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:08.089393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:56793)
0620 14:47:08.089532 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:08.089546 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:08.089619 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:08.089748 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:08.089749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:08.089937 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:08.089993 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:08.089995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:08.090775 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:08.090777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:08.091300 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:08.091301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:08.091695 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:08.091696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:08.091751 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:08.095515 (+ 3764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:08.095521 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:08.097208 (+ 1687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":58,"threads_started":1}
W20260620 14:47:08.097365 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:08.089488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56793 (local address 127.0.98.62:39179)
0620 14:47:08.089543 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:08.089545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:08.089611 (+ 66us) server_negotiation.cc:407] Connection header received
0620 14:47:08.089676 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:08.089677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:08.089688 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:08.089738 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:08.090049 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:08.090708 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:08.091372 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:08.091629 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:08.091680 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:08.097312 (+ 5632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56793: BlockingRecv error: recv got EOF from 127.0.98.2:56793 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 14:47:08 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:08 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:08.210423 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:08.205848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.3:43073)
0620 14:47:08.205960 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:08.205971 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:08.206042 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:08.206170 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:08.206171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:08.206369 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:08.206424 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:08.206426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:08.207098 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:08.207099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:08.208024 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:08.208027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:08.208360 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:08.208360 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:08.208412 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:08.208793 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:08.208797 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:08.210367 (+ 1570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
W20260620 14:47:08.210521 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:08.205909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43073 (local address 127.0.98.62:39179)
0620 14:47:08.205956 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:08.205958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:08.206023 (+ 65us) server_negotiation.cc:407] Connection header received
0620 14:47:08.206101 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:08.206102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:08.206114 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:08.206156 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:08.206490 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:08.207023 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:08.208094 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:08.208301 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:08.208355 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:08.210453 (+ 2098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43073: BlockingRecv error: recv got EOF from 127.0.98.3:43073 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 14:47:08 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:08 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:08.959261 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:08.954868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.1:57823)
0620 14:47:08.954995 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:08.955007 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:08.955058 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:08.955215 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:08.955216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:08.955365 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:08.955436 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:08.955438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:08.956104 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:08.956105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:08.956640 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:08.956642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:08.956989 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:08.956990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:08.957025 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:08.957503 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:08.957507 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:08.959186 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
W20260620 14:47:08.959337 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:08.954912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57823 (local address 127.0.98.62:39179)
0620 14:47:08.955033 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:08.955036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:08.955048 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:47:08.955120 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:08.955120 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:08.955132 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:08.955206 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:08.955510 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:08.956048 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:08.956729 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:08.956929 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:08.956992 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:08.959265 (+ 2273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57823: BlockingRecv error: recv got EOF from 127.0.98.1:57823 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
Jun 20 14:47:09 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:47:09 dist-test-slave-cx89 krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:47:09.102178 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:09.097918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39179 (local address 127.0.98.2:37741)
0620 14:47:09.098069 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:09.098082 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:09.098158 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:09.098289 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:09.098289 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:09.098506 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:09.098564 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:09.098566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:09.099179 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:09.099180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:09.099738 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:09.099740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:09.100039 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:09.100040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:09.100086 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:09.100525 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:09.100530 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:09.102111 (+ 1581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
W20260620 14:47:09.102243 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:09.098006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37741 (local address 127.0.98.62:39179)
0620 14:47:09.098068 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:09.098070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:09.098150 (+ 80us) server_negotiation.cc:407] Connection header received
0620 14:47:09.098213 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:09.098215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:09.098228 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:09.098280 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:09.098634 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:09.099111 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:09.099793 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:09.099975 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:09.100022 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:09.102185 (+ 2163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37741: BlockingRecv error: recv got EOF from 127.0.98.2:37741 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:47:09.129949 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 9126
I20260620 14:47:09.133932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 9277
I20260620 14:47:09.137492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 9410
I20260620 14:47:09.140962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 9056
2026-06-20T14:47:09Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17313 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T14:47:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:09Z Disabled control of system clock
I20260620 14:47:09.165479 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46699
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:47:09.242583 9607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:09.242792 9607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:09.242818 9607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:09.243964 9607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:09.244004 9607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:09.244019 9607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:09.244036 9607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:09.244050 9607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:47:09.245347 9607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46699
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:47:09.245573 9607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:09.245833 9607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:09.250458 9615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:09.250468 9613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:09.250494 9607 server_base.cc:1034] running on GCE node
W20260620 14:47:09.250474 9612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:09.250862 9607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:09.251109 9607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:09.252250 9607 hybrid_clock.cc:648] HybridClock initialized: now 1781966829252237 us; error 24 us; skew 500 ppm
I20260620 14:47:09.253350 9607 webserver.cc:460] Webserver started at http://127.0.98.62:42769/ using document root <none> and password file <none>
I20260620 14:47:09.253551 9607 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:09.253597 9607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:09.253717 9607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:09.254683 9607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "d21fddbfb70349248e1679143fcd68d8"
format_stamp: "Formatted at 2026-06-20 14:47:09 on dist-test-slave-cx89"
server_key: "2a7238198705a6beefe9a26741229a9f"
server_key_iv: "45d724e8be71aab7460ff2291625e542"
server_key_version: "encryptionkey@0"
I20260620 14:47:09.254993 9607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "d21fddbfb70349248e1679143fcd68d8"
format_stamp: "Formatted at 2026-06-20 14:47:09 on dist-test-slave-cx89"
server_key: "2a7238198705a6beefe9a26741229a9f"
server_key_iv: "45d724e8be71aab7460ff2291625e542"
server_key_version: "encryptionkey@0"
I20260620 14:47:09.256156 9607 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:47:09.256778 9621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:09.256909 9607 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:47:09.256984 9607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "d21fddbfb70349248e1679143fcd68d8"
format_stamp: "Formatted at 2026-06-20 14:47:09 on dist-test-slave-cx89"
server_key: "2a7238198705a6beefe9a26741229a9f"
server_key_iv: "45d724e8be71aab7460ff2291625e542"
server_key_version: "encryptionkey@0"
I20260620 14:47:09.257038 9607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:09.274119 9607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:09.274423 9607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:09.274544 9607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:09.278244 9607 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46699
I20260620 14:47:09.278266 9673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46699 every 8 connection(s)
I20260620 14:47:09.278589 9607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:09.279112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 9607
I20260620 14:47:09.279204 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:47:09.279302 9674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:09.279392 392 external_mini_cluster.cc:1428] Setting key 00581233ad2f8c94c5c3884d6b08b0b5
I20260620 14:47:09.282611 9674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8: Bootstrap starting.
I20260620 14:47:09.283216 9674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:09.283527 9674 log.cc:824] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:09.284217 9674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8: No bootstrap required, opened a new log
I20260620 14:47:09.284852 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:09.279911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38662 (local address 127.0.98.62:46699)
0620 14:47:09.280082 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:09.280089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:09.280113 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:47:09.280705 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:09.280712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:09.280778 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:09.280889 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:09.281126 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:09.281964 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:09.282998 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:09.283471 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:09.283565 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:09.284221 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:09.284225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:09.284225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:09.284414 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:09.284446 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:09.284457 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:09.284502 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:09.284592 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 14:47:09.284654 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260620 14:47:09.286130 9674 raft_consensus.cc:348] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21fddbfb70349248e1679143fcd68d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46699 } }
I20260620 14:47:09.286262 9674 raft_consensus.cc:374] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:09.286283 9674 raft_consensus.cc:729] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d21fddbfb70349248e1679143fcd68d8, State: Initialized, Role: FOLLOWER
I20260620 14:47:09.286407 9674 consensus_queue.cc:260] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21fddbfb70349248e1679143fcd68d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46699 } }
I20260620 14:47:09.286484 9674 raft_consensus.cc:388] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:09.286515 9674 raft_consensus.cc:482] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:09.286561 9674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:09.287070 9674 raft_consensus.cc:504] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21fddbfb70349248e1679143fcd68d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46699 } }
I20260620 14:47:09.287155 9674 leader_election.cc:302] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d21fddbfb70349248e1679143fcd68d8; no voters:
I20260620 14:47:09.287297 9674 leader_election.cc:288] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:09.287339 9679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:09.287506 9679 raft_consensus.cc:686] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8 [term 1 LEADER]: Becoming Leader. State: Replica: d21fddbfb70349248e1679143fcd68d8, State: Running, Role: LEADER
I20260620 14:47:09.287564 9674 sys_catalog.cc:564] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:09.287632 9679 consensus_queue.cc:237] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21fddbfb70349248e1679143fcd68d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46699 } }
I20260620 14:47:09.287993 9681 sys_catalog.cc:455] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d21fddbfb70349248e1679143fcd68d8. Latest consensus state: current_term: 1 leader_uuid: "d21fddbfb70349248e1679143fcd68d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21fddbfb70349248e1679143fcd68d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46699 } } }
I20260620 14:47:09.288008 9680 sys_catalog.cc:455] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d21fddbfb70349248e1679143fcd68d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21fddbfb70349248e1679143fcd68d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46699 } } }
I20260620 14:47:09.288143 9680 sys_catalog.cc:458] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:09.288130 9681 sys_catalog.cc:458] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:09.288687 9688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:09.289150 9688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:09.290498 9688 catalog_manager.cc:1347] Generated new cluster ID: d47290cf4d634c89b4e06f3a5db884d8
I20260620 14:47:09.290540 9688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:09.299652 9688 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:09.300137 9688 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:09.303611 9688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d21fddbfb70349248e1679143fcd68d8: Generated new TSK 0
I20260620 14:47:09.303759 9688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:47:09.305111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46699
--builtin_ntp_servers=127.0.98.20:46145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 14:47:09.418067 9698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:09.418412 9698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:09.418473 9698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:09.420142 9698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:09.420202 9698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:09.420249 9698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:47:09.422117 9698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:47:09.422372 9698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:09.422713 9698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:09.427651 9704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:09.427632 9706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:09.427636 9703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:09.427872 9698 server_base.cc:1034] running on GCE node
I20260620 14:47:09.428076 9698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:09.428341 9698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:09.429513 9698 hybrid_clock.cc:648] HybridClock initialized: now 1781966829429496 us; error 32 us; skew 500 ppm
I20260620 14:47:09.430910 9698 webserver.cc:460] Webserver started at http://127.0.98.1:44383/ using document root <none> and password file <none>
I20260620 14:47:09.431175 9698 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:09.431275 9698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:09.431437 9698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:09.432803 9698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "17793dc4af114631bc152c9c46d5e13b"
format_stamp: "Formatted at 2026-06-20 14:47:09 on dist-test-slave-cx89"
server_key: "53b34017b4f0a53ddb95b2e92dc12ee8"
server_key_iv: "08a805f925f80c5000fde5c6f88d957e"
server_key_version: "encryptionkey@0"
I20260620 14:47:09.433208 9698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "17793dc4af114631bc152c9c46d5e13b"
format_stamp: "Formatted at 2026-06-20 14:47:09 on dist-test-slave-cx89"
server_key: "53b34017b4f0a53ddb95b2e92dc12ee8"
server_key_iv: "08a805f925f80c5000fde5c6f88d957e"
server_key_version: "encryptionkey@0"
I20260620 14:47:09.434754 9698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:47:09.435560 9712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:09.435719 9698 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:47:09.435809 9698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "17793dc4af114631bc152c9c46d5e13b"
format_stamp: "Formatted at 2026-06-20 14:47:09 on dist-test-slave-cx89"
server_key: "53b34017b4f0a53ddb95b2e92dc12ee8"
server_key_iv: "08a805f925f80c5000fde5c6f88d957e"
server_key_version: "encryptionkey@0"
I20260620 14:47:09.435878 9698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:09.462532 9698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:09.462795 9698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:09.462893 9698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:09.463074 9698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:09.463366 9698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:09.463395 9698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:09.463442 9698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:09.463491 9698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:09.468802 9698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36057
I20260620 14:47:09.468859 9825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36057 every 8 connection(s)
I20260620 14:47:09.469144 9698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:47:09.470422 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 9698
I20260620 14:47:09.470485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:47:09.470649 392 external_mini_cluster.cc:1428] Setting key 79996a3d9eda8f17f1bf98c307eb04c2
I20260620 14:47:09.471506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46699
--builtin_ntp_servers=127.0.98.20:46145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:47:09.474128 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:09.469813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53159 (local address 127.0.98.62:46699)
0620 14:47:09.469870 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:09.469872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:09.470118 (+ 246us) server_negotiation.cc:407] Connection header received
0620 14:47:09.470642 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:09.470644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:09.470658 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:09.470711 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:09.471337 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:09.471936 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:09.472875 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:09.473072 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:09.473124 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:09.473896 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:09.473897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:09.473898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:09.473901 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:09.473917 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:09.473921 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:09.473964 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:09.474038 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 14:47:09.474069 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:47:09.474354 9828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:09.469811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46699 (local address 127.0.98.1:53159)
0620 14:47:09.470000 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:09.470033 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:09.470551 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:09.470782 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:09.470785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:09.470800 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:09.471260 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:09.471264 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:09.472017 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:09.472018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:09.472800 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:09.472802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:09.473136 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:09.473137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:09.473263 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:09.473750 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:47:09.473751 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:09.473789 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:47:09.473814 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:09.473973 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:09.473976 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:09.473979 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:47:09.474040 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:47:09.474148 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260620 14:47:09.474817 9826 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46699
I20260620 14:47:09.474903 9826 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:09.475073 9826 heartbeater.cc:511] Master 127.0.98.62:46699 requested a full tablet report, sending...
I20260620 14:47:09.475564 9638 ts_manager.cc:194] Registered new tserver with Master: 17793dc4af114631bc152c9c46d5e13b (127.0.98.1:36057)
I20260620 14:47:09.476152 9638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:53159
W20260620 14:47:09.549024 9829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:09.549228 9829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:09.549268 9829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:09.550384 9829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:09.550423 9829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:09.550455 9829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:47:09.551744 9829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:47:09.551965 9829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:09.552217 9829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:09.556861 9834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:09.556887 9837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:09.556891 9829 server_base.cc:1034] running on GCE node
W20260620 14:47:09.556856 9835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:09.557181 9829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:09.557396 9829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:09.558542 9829 hybrid_clock.cc:648] HybridClock initialized: now 1781966829558530 us; error 28 us; skew 500 ppm
I20260620 14:47:09.559849 9829 webserver.cc:460] Webserver started at http://127.0.98.2:45549/ using document root <none> and password file <none>
I20260620 14:47:09.560091 9829 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:09.560151 9829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:09.560288 9829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:09.561378 9829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "4a0714e269eb4b378c96e3c28a0ea12d"
format_stamp: "Formatted at 2026-06-20 14:47:09 on dist-test-slave-cx89"
server_key: "babd67f7c474be48b8a14abb78bcc44e"
server_key_iv: "5b52339731b8389c2bf00f278156e88b"
server_key_version: "encryptionkey@0"
I20260620 14:47:09.561728 9829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4a0714e269eb4b378c96e3c28a0ea12d"
format_stamp: "Formatted at 2026-06-20 14:47:09 on dist-test-slave-cx89"
server_key: "babd67f7c474be48b8a14abb78bcc44e"
server_key_iv: "5b52339731b8389c2bf00f278156e88b"
server_key_version: "encryptionkey@0"
I20260620 14:47:09.562992 9829 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:47:09.563737 9843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:09.563889 9829 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:09.563975 9829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "4a0714e269eb4b378c96e3c28a0ea12d"
format_stamp: "Formatted at 2026-06-20 14:47:09 on dist-test-slave-cx89"
server_key: "babd67f7c474be48b8a14abb78bcc44e"
server_key_iv: "5b52339731b8389c2bf00f278156e88b"
server_key_version: "encryptionkey@0"
I20260620 14:47:09.564044 9829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:09.575381 9829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:09.575742 9829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:09.575884 9829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:09.576113 9829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:09.576471 9829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:09.576508 9829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:09.576534 9829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:09.576552 9829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:09.582513 9829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39093
I20260620 14:47:09.582561 9956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39093 every 8 connection(s)
I20260620 14:47:09.582888 9829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:47:09.586447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 9829
I20260620 14:47:09.586542 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:47:09.586736 392 external_mini_cluster.cc:1428] Setting key 90974dddee5e9462928b60915296ee64
I20260620 14:47:09.587740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46699
--builtin_ntp_servers=127.0.98.20:46145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:47:09.588133 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:09.583604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35717 (local address 127.0.98.62:46699)
0620 14:47:09.583667 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:09.583669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:09.583878 (+ 209us) server_negotiation.cc:407] Connection header received
0620 14:47:09.584396 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:09.584398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:09.584408 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:09.584470 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:09.585055 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:09.585530 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:09.586487 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:09.586795 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:09.586891 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:09.587798 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:09.587799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:09.587800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:09.587805 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:09.587824 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:09.587829 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:09.587892 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:09.588000 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 14:47:09.588026 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:47:09.588289 9959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:09.583586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46699 (local address 127.0.98.2:35717)
0620 14:47:09.583767 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:09.583807 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:09.584294 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:09.584525 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:09.584528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:09.584536 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:09.584995 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:09.584999 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:09.585599 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:09.585600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:09.586395 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:09.586397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:09.586940 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:09.586941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:09.587096 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:09.587663 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:47:09.587663 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:09.587694 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:47:09.587717 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:09.587922 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:09.587927 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:09.587930 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:47:09.587988 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:47:09.588078 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260620 14:47:09.588714 9957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46699
I20260620 14:47:09.588799 9957 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:09.588982 9957 heartbeater.cc:511] Master 127.0.98.62:46699 requested a full tablet report, sending...
I20260620 14:47:09.589352 9638 ts_manager.cc:194] Registered new tserver with Master: 4a0714e269eb4b378c96e3c28a0ea12d (127.0.98.2:39093)
I20260620 14:47:09.589709 9638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35717
W20260620 14:47:09.665488 9960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:09.665683 9960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:09.665702 9960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:09.666862 9960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:09.666899 9960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:09.666925 9960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:47:09.668262 9960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:47:09.668493 9960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:09.668754 9960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:09.673256 9966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:09.673352 9960 server_base.cc:1034] running on GCE node
W20260620 14:47:09.673264 9965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:09.673278 9968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:09.673606 9960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:09.673836 9960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:09.674965 9960 hybrid_clock.cc:648] HybridClock initialized: now 1781966829674945 us; error 33 us; skew 500 ppm
I20260620 14:47:09.676162 9960 webserver.cc:460] Webserver started at http://127.0.98.3:34821/ using document root <none> and password file <none>
I20260620 14:47:09.676383 9960 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:09.676427 9960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:09.676560 9960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:09.677554 9960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "fa56f295022d40a3901665aeef7ab815"
format_stamp: "Formatted at 2026-06-20 14:47:09 on dist-test-slave-cx89"
server_key: "48bc17afa0948defacbfa62aaeccb3db"
server_key_iv: "8cc279b8755138f4ff373d205d5bd743"
server_key_version: "encryptionkey@0"
I20260620 14:47:09.677872 9960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fa56f295022d40a3901665aeef7ab815"
format_stamp: "Formatted at 2026-06-20 14:47:09 on dist-test-slave-cx89"
server_key: "48bc17afa0948defacbfa62aaeccb3db"
server_key_iv: "8cc279b8755138f4ff373d205d5bd743"
server_key_version: "encryptionkey@0"
I20260620 14:47:09.679032 9960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:47:09.680460 9974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:09.680622 9960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:47:09.680699 9960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "fa56f295022d40a3901665aeef7ab815"
format_stamp: "Formatted at 2026-06-20 14:47:09 on dist-test-slave-cx89"
server_key: "48bc17afa0948defacbfa62aaeccb3db"
server_key_iv: "8cc279b8755138f4ff373d205d5bd743"
server_key_version: "encryptionkey@0"
I20260620 14:47:09.680765 9960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:09.701799 9960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:09.702100 9960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:09.702225 9960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:09.702428 9960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:09.702750 9960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:09.702783 9960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:09.702811 9960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:09.702842 9960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:09.708103 9960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41053
I20260620 14:47:09.708158 10087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41053 every 8 connection(s)
I20260620 14:47:09.708449 9960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:47:09.712486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 9960
I20260620 14:47:09.712567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:47:09.712733 392 external_mini_cluster.cc:1428] Setting key 62963d858abea7c586958c0084e699f1
I20260620 14:47:09.713446 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:09.709124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42933 (local address 127.0.98.62:46699)
0620 14:47:09.709185 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:09.709187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:09.709424 (+ 237us) server_negotiation.cc:407] Connection header received
0620 14:47:09.709945 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:09.709947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:09.709959 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:09.710012 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:09.710670 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:09.711153 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:09.712133 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:09.712350 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:09.712416 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:09.713190 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:09.713191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:09.713192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:09.713197 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:09.713212 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:09.713216 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:09.713264 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:09.713345 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 14:47:09.713369 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:47:09.713636 10090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:09.709105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46699 (local address 127.0.98.3:42933)
0620 14:47:09.709293 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:09.709333 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:09.709842 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:09.710086 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:09.710089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:09.710097 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:09.710578 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:09.710582 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:09.711236 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:09.711237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:09.712048 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:09.712050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:09.712437 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:09.712439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:09.712560 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:09.713078 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:47:09.713079 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:09.713106 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:47:09.713127 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:09.713272 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:09.713276 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:09.713279 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:47:09.713347 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:47:09.713428 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260620 14:47:09.714059 10088 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46699
I20260620 14:47:09.714133 10088 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:09.714265 10088 heartbeater.cc:511] Master 127.0.98.62:46699 requested a full tablet report, sending...
I20260620 14:47:09.714541 9638 ts_manager.cc:194] Registered new tserver with Master: fa56f295022d40a3901665aeef7ab815 (127.0.98.3:41053)
I20260620 14:47:09.714887 9638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:42933
I20260620 14:47:09.724431 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:47:09.726071 9677 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:09.725658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.0.98.62:46699)
0620 14:47:09.725703 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:09.725704 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:09.725797 (+ 93us) server_negotiation.cc:407] Connection header received
0620 14:47:09.725844 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:09.725845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:09.725853 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:09.725902 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:09.726009 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38676: BlockingRecv error: recv got EOF from 127.0.0.1:38676 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:47:09.726162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 9698
I20260620 14:47:09.730463 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 9829
I20260620 14:47:09.733791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 9960
I20260620 14:47:09.737356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 9607
2026-06-20T14:47:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (595 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T14:47:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:09Z Disabled control of system clock
I20260620 14:47:09.761107 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43721
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:47:09.837383 10099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:09.837581 10099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:09.837601 10099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:09.838702 10099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:09.838742 10099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:09.838753 10099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:09.838766 10099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:09.838789 10099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:47:09.840126 10099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:47:09.840353 10099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:09.840615 10099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:09.844940 10105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:09.845072 10099 server_base.cc:1034] running on GCE node
W20260620 14:47:09.844940 10107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:09.845026 10104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:09.845412 10099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:09.845664 10099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:09.846794 10099 hybrid_clock.cc:648] HybridClock initialized: now 1781966829846785 us; error 28 us; skew 500 ppm
I20260620 14:47:09.847955 10099 webserver.cc:460] Webserver started at http://127.0.98.62:39981/ using document root <none> and password file <none>
I20260620 14:47:09.848151 10099 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:09.848194 10099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:09.848315 10099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:09.849313 10099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "a299e7b10e2143f8b6072d4eec49b044"
format_stamp: "Formatted at 2026-06-20 14:47:09 on dist-test-slave-cx89"
server_key: "7b5d429494f1ed02e658b552531eda57"
server_key_iv: "4b01f966aeac691443633855219fc2ee"
server_key_version: "encryptionkey@0"
I20260620 14:47:09.849629 10099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "a299e7b10e2143f8b6072d4eec49b044"
format_stamp: "Formatted at 2026-06-20 14:47:09 on dist-test-slave-cx89"
server_key: "7b5d429494f1ed02e658b552531eda57"
server_key_iv: "4b01f966aeac691443633855219fc2ee"
server_key_version: "encryptionkey@0"
I20260620 14:47:09.850811 10099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:47:09.851440 10113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:09.851584 10099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:47:09.851660 10099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "a299e7b10e2143f8b6072d4eec49b044"
format_stamp: "Formatted at 2026-06-20 14:47:09 on dist-test-slave-cx89"
server_key: "7b5d429494f1ed02e658b552531eda57"
server_key_iv: "4b01f966aeac691443633855219fc2ee"
server_key_version: "encryptionkey@0"
I20260620 14:47:09.851723 10099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:09.854816 10099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:09.855072 10099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:09.855188 10099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:09.858680 10099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43721
I20260620 14:47:09.858709 10165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43721 every 8 connection(s)
I20260620 14:47:09.858989 10099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:09.859581 10166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:09.862083 10166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044: Bootstrap starting.
I20260620 14:47:09.862605 10166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:09.862860 10166 log.cc:824] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:09.863478 10166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044: No bootstrap required, opened a new log
I20260620 14:47:09.864573 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 10099
I20260620 14:47:09.864655 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:47:09.864813 392 external_mini_cluster.cc:1428] Setting key 517768bebedbc728cc729f787934f07d
I20260620 14:47:09.865051 10166 raft_consensus.cc:348] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a299e7b10e2143f8b6072d4eec49b044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43721 } }
I20260620 14:47:09.865222 10166 raft_consensus.cc:374] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:09.865267 10166 raft_consensus.cc:729] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a299e7b10e2143f8b6072d4eec49b044, State: Initialized, Role: FOLLOWER
I20260620 14:47:09.865435 10166 consensus_queue.cc:260] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a299e7b10e2143f8b6072d4eec49b044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43721 } }
I20260620 14:47:09.865502 10166 raft_consensus.cc:388] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:09.865525 10166 raft_consensus.cc:482] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:09.865568 10166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:09.866097 10166 raft_consensus.cc:504] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a299e7b10e2143f8b6072d4eec49b044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43721 } }
I20260620 14:47:09.866200 10166 leader_election.cc:302] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a299e7b10e2143f8b6072d4eec49b044; no voters:
I20260620 14:47:09.866360 10166 leader_election.cc:288] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:09.866401 10171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:09.866566 10171 raft_consensus.cc:686] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044 [term 1 LEADER]: Becoming Leader. State: Replica: a299e7b10e2143f8b6072d4eec49b044, State: Running, Role: LEADER
I20260620 14:47:09.866645 10171 consensus_queue.cc:237] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a299e7b10e2143f8b6072d4eec49b044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43721 } }
I20260620 14:47:09.866688 10166 sys_catalog.cc:564] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:09.866942 10171 sys_catalog.cc:455] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a299e7b10e2143f8b6072d4eec49b044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a299e7b10e2143f8b6072d4eec49b044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43721 } } }
I20260620 14:47:09.867008 10171 sys_catalog.cc:458] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:09.867110 10171 sys_catalog.cc:455] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a299e7b10e2143f8b6072d4eec49b044. Latest consensus state: current_term: 1 leader_uuid: "a299e7b10e2143f8b6072d4eec49b044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a299e7b10e2143f8b6072d4eec49b044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43721 } } }
I20260620 14:47:09.867233 10171 sys_catalog.cc:458] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:09.867244 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:09.865277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40672 (local address 127.0.98.62:43721)
0620 14:47:09.865476 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:09.865486 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:09.865518 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:47:09.865631 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:09.865638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:09.865679 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:09.865790 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:09.866447 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:09.866450 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:09.866451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:09.866454 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:09.866494 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:09.866506 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:09.866560 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:09.866714 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 14:47:09.866795 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260620 14:47:09.867729 10176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:09.868242 10176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:09.869539 10176 catalog_manager.cc:1347] Generated new cluster ID: 7cad3166a7d341ff89c55a7c7444428f
I20260620 14:47:09.869587 10176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:09.891306 10176 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:09.891497 10176 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:09.894860 10176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a299e7b10e2143f8b6072d4eec49b044: Generated new TSK 0
I20260620 14:47:09.895025 10176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:47:09.905320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43721
--builtin_ntp_servers=127.0.98.20:33763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:47:10.007869 10190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:10.008072 10190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:10.008101 10190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:10.009232 10190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:10.009269 10190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:10.009302 10190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:47:10.010635 10190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:47:10.010866 10190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:10.011142 10190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:10.015727 10196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:10.015753 10195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:10.015717 10198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:10.016340 10190 server_base.cc:1034] running on GCE node
I20260620 14:47:10.016543 10190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:10.016752 10190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:10.017879 10190 hybrid_clock.cc:648] HybridClock initialized: now 1781966830017864 us; error 30 us; skew 500 ppm
I20260620 14:47:10.018965 10190 webserver.cc:460] Webserver started at http://127.0.98.1:45277/ using document root <none> and password file <none>
I20260620 14:47:10.019171 10190 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:10.019218 10190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:10.019340 10190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:10.020412 10190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "71dd8001fc8a40418c0d404a46980271"
format_stamp: "Formatted at 2026-06-20 14:47:10 on dist-test-slave-cx89"
server_key: "4ddb2c6accc635b746545fb7b379b43c"
server_key_iv: "e29dbdc571013ac3a377a030ded8e880"
server_key_version: "encryptionkey@0"
I20260620 14:47:10.020735 10190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "71dd8001fc8a40418c0d404a46980271"
format_stamp: "Formatted at 2026-06-20 14:47:10 on dist-test-slave-cx89"
server_key: "4ddb2c6accc635b746545fb7b379b43c"
server_key_iv: "e29dbdc571013ac3a377a030ded8e880"
server_key_version: "encryptionkey@0"
I20260620 14:47:10.021888 10190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:47:10.022486 10204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:10.022635 10190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:47:10.022710 10190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "71dd8001fc8a40418c0d404a46980271"
format_stamp: "Formatted at 2026-06-20 14:47:10 on dist-test-slave-cx89"
server_key: "4ddb2c6accc635b746545fb7b379b43c"
server_key_iv: "e29dbdc571013ac3a377a030ded8e880"
server_key_version: "encryptionkey@0"
I20260620 14:47:10.022773 10190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:10.025949 10190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:10.026176 10190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:10.026289 10190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:10.026473 10190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:10.026733 10190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:10.026763 10190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:10.026788 10190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:10.026811 10190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:10.031960 10190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33609
I20260620 14:47:10.032014 10317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33609 every 8 connection(s)
I20260620 14:47:10.032274 10190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:47:10.034749 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:10.032899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59871 (local address 127.0.98.62:43721)
0620 14:47:10.032939 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:10.032941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:10.033151 (+ 210us) server_negotiation.cc:407] Connection header received
0620 14:47:10.033572 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:10.033573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:10.033581 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:10.033624 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:10.034580 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:10.034581 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:10.034581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:10.034584 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:10.034597 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:10.034602 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:10.034641 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:10.034687 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 14:47:10.034699 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:47:10.034926 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:10.032891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43721 (local address 127.0.98.1:59871)
0620 14:47:10.033046 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:10.033077 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:10.033500 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:10.033683 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:10.033686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:10.033693 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:10.034463 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:47:10.034463 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:10.034488 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:47:10.034514 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:10.034640 (+ 126us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:10.034642 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:10.034645 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:47:10.034693 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:47:10.034736 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260620 14:47:10.035351 10318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43721
I20260620 14:47:10.035460 10318 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:10.035585 10318 heartbeater.cc:511] Master 127.0.98.62:43721 requested a full tablet report, sending...
I20260620 14:47:10.036039 10130 ts_manager.cc:194] Registered new tserver with Master: 71dd8001fc8a40418c0d404a46980271 (127.0.98.1:33609)
I20260620 14:47:10.040680 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 10190
I20260620 14:47:10.040755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:47:10.040913 392 external_mini_cluster.cc:1428] Setting key 67f10640e6ec1f9d6c7e759d99539e16
I20260620 14:47:10.041925 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43721
--builtin_ntp_servers=127.0.98.20:33763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:47:10.118441 10321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:10.118638 10321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:10.118659 10321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:10.119747 10321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:10.119781 10321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:10.119812 10321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:47:10.121058 10321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:47:10.121239 10321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:10.121469 10321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:10.126104 10329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:10.126091 10327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:10.126068 10321 server_base.cc:1034] running on GCE node
W20260620 14:47:10.126149 10326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:10.126467 10321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:10.126703 10321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:10.127836 10321 hybrid_clock.cc:648] HybridClock initialized: now 1781966830127823 us; error 29 us; skew 500 ppm
I20260620 14:47:10.128952 10321 webserver.cc:460] Webserver started at http://127.0.98.2:37851/ using document root <none> and password file <none>
I20260620 14:47:10.129163 10321 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:10.129209 10321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:10.129339 10321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:10.130328 10321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "cc9a3db282654a59b80ce450b1afbe39"
format_stamp: "Formatted at 2026-06-20 14:47:10 on dist-test-slave-cx89"
server_key: "249023dc39b3819381fb7dce305aba5d"
server_key_iv: "09bb8db4577dbca7fa357da1d61fe3e9"
server_key_version: "encryptionkey@0"
I20260620 14:47:10.130650 10321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cc9a3db282654a59b80ce450b1afbe39"
format_stamp: "Formatted at 2026-06-20 14:47:10 on dist-test-slave-cx89"
server_key: "249023dc39b3819381fb7dce305aba5d"
server_key_iv: "09bb8db4577dbca7fa357da1d61fe3e9"
server_key_version: "encryptionkey@0"
I20260620 14:47:10.131903 10321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:47:10.132544 10335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:10.132691 10321 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:47:10.132767 10321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "cc9a3db282654a59b80ce450b1afbe39"
format_stamp: "Formatted at 2026-06-20 14:47:10 on dist-test-slave-cx89"
server_key: "249023dc39b3819381fb7dce305aba5d"
server_key_iv: "09bb8db4577dbca7fa357da1d61fe3e9"
server_key_version: "encryptionkey@0"
I20260620 14:47:10.132828 10321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:10.136068 10321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:10.136304 10321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:10.136428 10321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:10.136618 10321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:10.136894 10321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:10.136926 10321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:10.136953 10321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:10.136976 10321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:10.142225 10321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41023
I20260620 14:47:10.142273 10448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41023 every 8 connection(s)
I20260620 14:47:10.142561 10321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:47:10.145315 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:10.143181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35585 (local address 127.0.98.62:43721)
0620 14:47:10.143233 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:10.143236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:10.143460 (+ 224us) server_negotiation.cc:407] Connection header received
0620 14:47:10.143934 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:10.143935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:10.143942 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:10.143989 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:10.145081 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:10.145082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:10.145083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:10.145087 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:10.145104 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:10.145109 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:10.145157 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:10.145226 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:47:10.145240 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:47:10.145493 10451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:10.143175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43721 (local address 127.0.98.2:35585)
0620 14:47:10.143331 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:10.143364 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:10.143840 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:10.144065 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:10.144069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:10.144077 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:10.144944 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:47:10.144944 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:10.144972 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:47:10.144999 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:10.145158 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:10.145163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:10.145166 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:47:10.145233 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:47:10.145292 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
I20260620 14:47:10.145913 10449 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43721
I20260620 14:47:10.145987 10449 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:10.146080 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 10321
I20260620 14:47:10.146118 10449 heartbeater.cc:511] Master 127.0.98.62:43721 requested a full tablet report, sending...
I20260620 14:47:10.146166 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:47:10.146351 392 external_mini_cluster.cc:1428] Setting key 0eba09f61399abb9abd157e41a709077
I20260620 14:47:10.146443 10130 ts_manager.cc:194] Registered new tserver with Master: cc9a3db282654a59b80ce450b1afbe39 (127.0.98.2:41023)
I20260620 14:47:10.147181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43721
--builtin_ntp_servers=127.0.98.20:33763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:47:10.224253 10452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:10.224524 10452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:10.224556 10452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:10.226032 10452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:10.226083 10452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:10.226117 10452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:47:10.227646 10452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:47:10.227859 10452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:10.228118 10452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:47:10.232851 10452 server_base.cc:1034] running on GCE node
W20260620 14:47:10.232872 10458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:10.232978 10457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:10.232919 10460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:10.233301 10452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:10.233536 10452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:10.234668 10452 hybrid_clock.cc:648] HybridClock initialized: now 1781966830234651 us; error 30 us; skew 500 ppm
I20260620 14:47:10.235847 10452 webserver.cc:460] Webserver started at http://127.0.98.3:35917/ using document root <none> and password file <none>
I20260620 14:47:10.236073 10452 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:10.236120 10452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:10.236248 10452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:10.237267 10452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "b408315807e34ffca1ff6754ed27e28a"
format_stamp: "Formatted at 2026-06-20 14:47:10 on dist-test-slave-cx89"
server_key: "544de83fb1c8153bf06a4e228125ae67"
server_key_iv: "4f77d7786f05cab47e4eb59cee40ac21"
server_key_version: "encryptionkey@0"
I20260620 14:47:10.237596 10452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b408315807e34ffca1ff6754ed27e28a"
format_stamp: "Formatted at 2026-06-20 14:47:10 on dist-test-slave-cx89"
server_key: "544de83fb1c8153bf06a4e228125ae67"
server_key_iv: "4f77d7786f05cab47e4eb59cee40ac21"
server_key_version: "encryptionkey@0"
I20260620 14:47:10.238719 10452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:47:10.239297 10466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:10.239454 10452 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:47:10.239535 10452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "b408315807e34ffca1ff6754ed27e28a"
format_stamp: "Formatted at 2026-06-20 14:47:10 on dist-test-slave-cx89"
server_key: "544de83fb1c8153bf06a4e228125ae67"
server_key_iv: "4f77d7786f05cab47e4eb59cee40ac21"
server_key_version: "encryptionkey@0"
I20260620 14:47:10.239599 10452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:10.242666 10452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:10.242892 10452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:10.243002 10452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:10.243176 10452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:10.243466 10452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:10.243505 10452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:10.243531 10452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:10.243554 10452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:10.248767 10452 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44667
I20260620 14:47:10.248805 10579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44667 every 8 connection(s)
I20260620 14:47:10.249095 10452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:47:10.251194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 10452
I20260620 14:47:10.251286 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:47:10.251525 392 external_mini_cluster.cc:1428] Setting key 7e67c2159be23f11da406408ab0f844d
I20260620 14:47:10.251982 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:10.249720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52807 (local address 127.0.98.62:43721)
0620 14:47:10.249766 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:10.249767 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:10.249987 (+ 220us) server_negotiation.cc:407] Connection header received
0620 14:47:10.250442 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:10.250444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:10.250450 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:10.250508 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:10.251722 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:10.251724 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:10.251724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:10.251728 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:10.251742 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:10.251747 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:10.251801 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:10.251903 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 14:47:10.251917 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:47:10.252180 10582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:10.249700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43721 (local address 127.0.98.3:52807)
0620 14:47:10.249881 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:10.249911 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:10.250352 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:10.250579 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:10.250584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:10.250593 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:10.251564 (+ 971us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:47:10.251565 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:10.251612 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:47:10.251646 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:10.251815 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:10.251822 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:10.251825 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:47:10.251909 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:47:10.251971 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260620 14:47:10.252569 10580 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43721
I20260620 14:47:10.252646 10580 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:10.252779 10580 heartbeater.cc:511] Master 127.0.98.62:43721 requested a full tablet report, sending...
I20260620 14:47:10.253075 10130 ts_manager.cc:194] Registered new tserver with Master: b408315807e34ffca1ff6754ed27e28a (127.0.98.3:44667)
I20260620 14:47:10.263185 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:47:10.264940 10170 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:10.264494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40684 (local address 127.0.98.62:43721)
0620 14:47:10.264565 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:10.264567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:10.264625 (+ 58us) server_negotiation.cc:407] Connection header received
0620 14:47:10.264696 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:10.264697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:10.264701 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:10.264747 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:10.264876 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40684: BlockingRecv error: recv got EOF from 127.0.0.1:40684 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:47:10.264989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 10190
I20260620 14:47:10.269486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 10321
I20260620 14:47:10.273778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 10452
I20260620 14:47:10.277096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 10099
2026-06-20T14:47:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (539 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:47:10 dist-test-slave-cx89 krb5kdc[10590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:47:10 dist-test-slave-cx89 krb5kdc[10590](info): set up 2 sockets
Jun 20 14:47:10 dist-test-slave-cx89 krb5kdc[10590](info): commencing operation
krb5kdc: starting...
W20260620 14:47:11.223400 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:47:11 dist-test-slave-cx89 krb5kdc[10590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:47:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:47:11.331362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35735
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:11.407883 10603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:11.408074 10603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:11.408094 10603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:11.409196 10603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:11.409236 10603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:11.409250 10603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:11.409260 10603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:11.409271 10603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:47:11.410697 10603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:47:11.410913 10603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:11.411118 10603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:11.415560 10611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:11.415592 10609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:11.415596 10608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:11.415640 10603 server_base.cc:1034] running on GCE node
I20260620 14:47:11.415974 10603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:11.416224 10603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:11.417354 10603 hybrid_clock.cc:648] HybridClock initialized: now 1781966831417345 us; error 28 us; skew 500 ppm
Jun 20 14:47:11 dist-test-slave-cx89 krb5kdc[10590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:11.419592 10603 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:47:11.420004 10603 webserver.cc:460] Webserver started at http://127.0.98.62:45925/ using document root <none> and password file <none>
I20260620 14:47:11.420192 10603 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:11.420234 10603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:11.420351 10603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:11.421340 10603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "966becd32e9447d0819095951f49084c"
format_stamp: "Formatted at 2026-06-20 14:47:11 on dist-test-slave-cx89"
server_key: "8a0dd6a6cc12cec124b60e5fccb21367"
server_key_iv: "1d17154a075348b6a47d20c8231f9cd9"
server_key_version: "encryptionkey@0"
I20260620 14:47:11.421650 10603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "966becd32e9447d0819095951f49084c"
format_stamp: "Formatted at 2026-06-20 14:47:11 on dist-test-slave-cx89"
server_key: "8a0dd6a6cc12cec124b60e5fccb21367"
server_key_iv: "1d17154a075348b6a47d20c8231f9cd9"
server_key_version: "encryptionkey@0"
I20260620 14:47:11.422808 10603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:47:11.423534 10618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:11.423661 10603 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:47:11.423735 10603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "966becd32e9447d0819095951f49084c"
format_stamp: "Formatted at 2026-06-20 14:47:11 on dist-test-slave-cx89"
server_key: "8a0dd6a6cc12cec124b60e5fccb21367"
server_key_iv: "1d17154a075348b6a47d20c8231f9cd9"
server_key_version: "encryptionkey@0"
I20260620 14:47:11.423796 10603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:11.442142 10603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:11.444036 10603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:11.444154 10603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:11.447662 10603 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35735
I20260620 14:47:11.447703 10670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35735 every 8 connection(s)
I20260620 14:47:11.448012 10603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:11.448885 10671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:11.451009 10671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c: Bootstrap starting.
I20260620 14:47:11.451550 10671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:11.451771 10671 log.cc:824] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:11.452358 10671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c: No bootstrap required, opened a new log
I20260620 14:47:11.453840 10671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966becd32e9447d0819095951f49084c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35735 } }
I20260620 14:47:11.453972 10671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:11.453994 10671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 966becd32e9447d0819095951f49084c, State: Initialized, Role: FOLLOWER
I20260620 14:47:11.454092 10671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966becd32e9447d0819095951f49084c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35735 } }
I20260620 14:47:11.454142 10671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:11.454169 10671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:11.454205 10671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:11.454928 10671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966becd32e9447d0819095951f49084c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35735 } }
I20260620 14:47:11.455017 10671 leader_election.cc:302] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 966becd32e9447d0819095951f49084c; no voters:
I20260620 14:47:11.455156 10671 leader_election.cc:288] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:11.455191 10674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:11.455333 10674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c [term 1 LEADER]: Becoming Leader. State: Replica: 966becd32e9447d0819095951f49084c, State: Running, Role: LEADER
I20260620 14:47:11.455382 10671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:11.455441 10674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966becd32e9447d0819095951f49084c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35735 } }
I20260620 14:47:11.455600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 10603
I20260620 14:47:11.455685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:47:11.455873 392 external_mini_cluster.cc:1428] Setting key a027fc8ce638e4eb0e9c2475e698394d
I20260620 14:47:11.456091 10675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "966becd32e9447d0819095951f49084c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966becd32e9447d0819095951f49084c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35735 } } }
I20260620 14:47:11.456159 10675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:11.456432 10684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:11.456405 10676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 966becd32e9447d0819095951f49084c. Latest consensus state: current_term: 1 leader_uuid: "966becd32e9447d0819095951f49084c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966becd32e9447d0819095951f49084c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35735 } } }
I20260620 14:47:11.456593 10676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:11.456974 10684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:11.458274 10684 catalog_manager.cc:1347] Generated new cluster ID: 235c89407f314fc5a71cd212dc81e52a
I20260620 14:47:11.458335 10684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:47:11 dist-test-slave-cx89 krb5kdc[10590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:47:11.466882 10684 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:11.467536 10684 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:11.467934 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:11.456502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.0.98.62:35735)
0620 14:47:11.457727 (+ 1225us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:11.457734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:11.457760 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:47:11.457907 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:11.457915 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:11.457971 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:11.458083 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:11.458770 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.460116 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.461118 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.461738 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.461844 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:11.464446 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:11.464450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:11.464450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:11.464461 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:11.466302 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:11.466678 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:11.466681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:11.466681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:11.466785 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:11.467045 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:11.467046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:11.467046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:11.467294 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:11.467464 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:11.467573 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 14:47:11.467718 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1173,"thread_start_us":57,"threads_started":1}
I20260620 14:47:11.481338 10684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c: Generated new TSK 0
I20260620 14:47:11.481570 10684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:47:11.510192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35735
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:11.614035 10697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:11.614224 10697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:11.614246 10697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:11.615352 10697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:11.615391 10697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:11.615439 10697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:47:11.616848 10697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:47:11.617071 10697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:11.617262 10697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:11.621798 10703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:11.621930 10697 server_base.cc:1034] running on GCE node
W20260620 14:47:11.621904 10702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:11.621793 10705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:11.622228 10697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:11.622444 10697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:11.623579 10697 hybrid_clock.cc:648] HybridClock initialized: now 1781966831623562 us; error 32 us; skew 500 ppm
Jun 20 14:47:11 dist-test-slave-cx89 krb5kdc[10590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:11.625875 10697 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:47:11.626255 10697 webserver.cc:460] Webserver started at http://127.0.98.1:42015/ using document root <none> and password file <none>
I20260620 14:47:11.626454 10697 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:11.626497 10697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:11.626578 10697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:11.627617 10697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "31e612e23a67415dad338f8f3d2197a8"
format_stamp: "Formatted at 2026-06-20 14:47:11 on dist-test-slave-cx89"
server_key: "3a0f122f9c31d2adf84d2ee10079e2a8"
server_key_iv: "e45942762cebc42173996917350702c9"
server_key_version: "encryptionkey@0"
I20260620 14:47:11.627904 10697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "31e612e23a67415dad338f8f3d2197a8"
format_stamp: "Formatted at 2026-06-20 14:47:11 on dist-test-slave-cx89"
server_key: "3a0f122f9c31d2adf84d2ee10079e2a8"
server_key_iv: "e45942762cebc42173996917350702c9"
server_key_version: "encryptionkey@0"
I20260620 14:47:11.629045 10697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:47:11.629683 10712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:11.629858 10697 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:47:11.629945 10697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "31e612e23a67415dad338f8f3d2197a8"
format_stamp: "Formatted at 2026-06-20 14:47:11 on dist-test-slave-cx89"
server_key: "3a0f122f9c31d2adf84d2ee10079e2a8"
server_key_iv: "e45942762cebc42173996917350702c9"
server_key_version: "encryptionkey@0"
I20260620 14:47:11.629997 10697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:11.656291 10697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:11.658443 10697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:11.658576 10697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:11.658782 10697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:11.659075 10697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:11.659106 10697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:11.659128 10697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:11.659152 10697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:11.664621 10697 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36957
I20260620 14:47:11.664678 10825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36957 every 8 connection(s)
I20260620 14:47:11.664968 10697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:47:11.665151 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 10697
I20260620 14:47:11.665233 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:47:11.665442 392 external_mini_cluster.cc:1428] Setting key 10253805b61bf887d26704cb2a53c882
Jun 20 14:47:11 dist-test-slave-cx89 krb5kdc[10590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:47:11.674342 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:11.665777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56161 (local address 127.0.98.62:35735)
0620 14:47:11.665826 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:11.665828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:11.666034 (+ 206us) server_negotiation.cc:407] Connection header received
0620 14:47:11.666545 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:11.666547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:11.666559 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:11.666625 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:11.667709 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.668275 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.669227 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.669543 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.669616 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:11.671760 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:11.671762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:11.671762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:11.671770 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:11.673218 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:11.673571 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:11.673572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:11.673572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:11.673618 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:11.673793 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:11.673793 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:11.673793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:11.673896 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:11.673951 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:11.674210 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 14:47:11.674276 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:47:11.674506 10828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:11.665771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35735 (local address 127.0.98.1:56161)
0620 14:47:11.665924 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:11.665964 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:11.666456 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:11.666682 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:11.666685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:11.667328 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:11.667627 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:11.667634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.668357 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.668358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:11.669155 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:11.669157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.669614 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.669615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:11.669712 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:11.670181 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:11.670191 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:11.671675 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:11.673293 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:11.673300 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:11.673302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:11.673517 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:11.673659 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:11.673660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:11.673660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:11.673751 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:11.673956 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:11.673961 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:11.674107 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 14:47:11.674194 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:47:11.674313 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260620 14:47:11.674996 10826 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35735
I20260620 14:47:11.675091 10826 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:11.675258 10826 heartbeater.cc:511] Master 127.0.98.62:35735 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:47:11.675813 10635 ts_manager.cc:194] Registered new tserver with Master: 31e612e23a67415dad338f8f3d2197a8 (127.0.98.1:36957)
I20260620 14:47:11.676409 10635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:47:11.688179 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35735
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:11.766760 10831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:11.766958 10831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:11.766979 10831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:11.768129 10831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:11.768169 10831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:11.768194 10831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:47:11.769577 10831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:47:11.769763 10831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:11.769937 10831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:47:11.774633 10831 server_base.cc:1034] running on GCE node
W20260620 14:47:11.774590 10836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:11.774626 10839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:11.774561 10837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:11.774946 10831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:11.775175 10831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:11.776338 10831 hybrid_clock.cc:648] HybridClock initialized: now 1781966831776316 us; error 27 us; skew 500 ppm
Jun 20 14:47:11 dist-test-slave-cx89 krb5kdc[10590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:11.778780 10831 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:47:11.779211 10831 webserver.cc:460] Webserver started at http://127.0.98.2:39853/ using document root <none> and password file <none>
I20260620 14:47:11.779462 10831 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:11.779517 10831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:11.779654 10831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:11.780696 10831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "049201653cde40d7bf36b114345edb96"
format_stamp: "Formatted at 2026-06-20 14:47:11 on dist-test-slave-cx89"
server_key: "749fda4524f0b0102ec880ca8a72e706"
server_key_iv: "f44edfe14f0874e297f7e997713bd4df"
server_key_version: "encryptionkey@0"
I20260620 14:47:11.781051 10831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "049201653cde40d7bf36b114345edb96"
format_stamp: "Formatted at 2026-06-20 14:47:11 on dist-test-slave-cx89"
server_key: "749fda4524f0b0102ec880ca8a72e706"
server_key_iv: "f44edfe14f0874e297f7e997713bd4df"
server_key_version: "encryptionkey@0"
I20260620 14:47:11.782368 10831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:47:11.783038 10846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:11.783183 10831 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:47:11.783265 10831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "049201653cde40d7bf36b114345edb96"
format_stamp: "Formatted at 2026-06-20 14:47:11 on dist-test-slave-cx89"
server_key: "749fda4524f0b0102ec880ca8a72e706"
server_key_iv: "f44edfe14f0874e297f7e997713bd4df"
server_key_version: "encryptionkey@0"
I20260620 14:47:11.783342 10831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:11.821792 10831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:11.824057 10831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:11.824200 10831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:11.824436 10831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:11.824743 10831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:11.824776 10831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:11.824810 10831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:11.824828 10831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:11.830832 10831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43681
I20260620 14:47:11.830875 10959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43681 every 8 connection(s)
I20260620 14:47:11.831190 10831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:47:11.832698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 10831
I20260620 14:47:11.832779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:47:11.832950 392 external_mini_cluster.cc:1428] Setting key 5eb5f06f0eda9a3a04e2aae0a058cd2c
Jun 20 14:47:11 dist-test-slave-cx89 krb5kdc[10590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:47:11.841238 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:11.831958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46183 (local address 127.0.98.62:35735)
0620 14:47:11.832009 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:11.832012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:11.832229 (+ 217us) server_negotiation.cc:407] Connection header received
0620 14:47:11.832796 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:11.832797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:11.832811 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:11.832867 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:11.834035 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.834717 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.836071 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.836259 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.836320 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:11.838649 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:11.838651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:11.838651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:11.838657 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:11.839895 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:11.840306 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:11.840307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:11.840307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:11.840350 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:11.840566 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:11.840567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:11.840567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:11.840653 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:11.840706 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:11.841067 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 14:47:11.841145 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:47:11.841562 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:11.831941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35735 (local address 127.0.98.2:46183)
0620 14:47:11.832116 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:11.832158 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:11.832674 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:11.833037 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:11.833043 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:11.833736 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:11.833944 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:11.833949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.834792 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.834793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:11.835996 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:11.835998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.836339 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.836340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:11.836474 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:11.836966 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:11.836975 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:11.838542 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:11.839972 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:11.839978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:11.839980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:11.840240 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:11.840386 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:11.840387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:11.840387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:11.840513 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:11.840737 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:11.840744 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:11.840927 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 14:47:11.841039 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:47:11.841206 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260620 14:47:11.841996 10960 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35735
I20260620 14:47:11.842073 10960 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:11.842231 10960 heartbeater.cc:511] Master 127.0.98.62:35735 requested a full tablet report, sending...
I20260620 14:47:11.842576 10635 ts_manager.cc:194] Registered new tserver with Master: 049201653cde40d7bf36b114345edb96 (127.0.98.2:43681)
I20260620 14:47:11.842916 10635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:47:11.856293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35735
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:11.932555 10965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:11.932776 10965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:11.932797 10965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:11.933905 10965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:11.933948 10965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:11.933974 10965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:47:11.935329 10965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:47:11.935534 10965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:11.935709 10965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:11.940263 10970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:11.940286 10965 server_base.cc:1034] running on GCE node
W20260620 14:47:11.940239 10971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:11.940234 10973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:11.940631 10965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:11.940838 10965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:11.941967 10965 hybrid_clock.cc:648] HybridClock initialized: now 1781966831941954 us; error 25 us; skew 500 ppm
Jun 20 14:47:11 dist-test-slave-cx89 krb5kdc[10590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:11.944309 10965 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:47:11.944693 10965 webserver.cc:460] Webserver started at http://127.0.98.3:45891/ using document root <none> and password file <none>
I20260620 14:47:11.944882 10965 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:11.944931 10965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:11.945051 10965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:11.945998 10965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "03ba8a1beb2047e4bebca6d6e806fb5d"
format_stamp: "Formatted at 2026-06-20 14:47:11 on dist-test-slave-cx89"
server_key: "9a3fb5c67fcd1e67d57d8fbd177dc4fd"
server_key_iv: "c019aee46b69cf3be2239866f652de8c"
server_key_version: "encryptionkey@0"
I20260620 14:47:11.946305 10965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "03ba8a1beb2047e4bebca6d6e806fb5d"
format_stamp: "Formatted at 2026-06-20 14:47:11 on dist-test-slave-cx89"
server_key: "9a3fb5c67fcd1e67d57d8fbd177dc4fd"
server_key_iv: "c019aee46b69cf3be2239866f652de8c"
server_key_version: "encryptionkey@0"
I20260620 14:47:11.947480 10965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:47:11.948125 10980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:11.948266 10965 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:47:11.948351 10965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "03ba8a1beb2047e4bebca6d6e806fb5d"
format_stamp: "Formatted at 2026-06-20 14:47:11 on dist-test-slave-cx89"
server_key: "9a3fb5c67fcd1e67d57d8fbd177dc4fd"
server_key_iv: "c019aee46b69cf3be2239866f652de8c"
server_key_version: "encryptionkey@0"
I20260620 14:47:11.948405 10965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:11.965669 10965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:11.967731 10965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:11.967856 10965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:11.968050 10965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:11.968325 10965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:11.968361 10965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:11.968389 10965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:11.968412 10965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:11.973650 10965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41781
I20260620 14:47:11.973711 11093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41781 every 8 connection(s)
I20260620 14:47:11.973979 10965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:47:11 dist-test-slave-cx89 krb5kdc[10590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:47:11.980401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 10965
I20260620 14:47:11.980494 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:47:11.980638 392 external_mini_cluster.cc:1428] Setting key b0159fec55e7344dff57a5973d57eed7
I20260620 14:47:11.982452 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:11.974600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51897 (local address 127.0.98.62:35735)
0620 14:47:11.974641 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:11.974643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:11.974834 (+ 191us) server_negotiation.cc:407] Connection header received
0620 14:47:11.975316 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:11.975318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:11.975328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:11.975384 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:11.976270 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.976763 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.977665 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.977848 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.977892 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:11.979761 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:11.979762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:11.979763 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:11.979769 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:11.981185 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:11.981606 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:11.981607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:11.981608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:11.981665 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:11.981885 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:11.981886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:11.981886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:11.981993 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:11.982047 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:11.982292 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 14:47:11.982365 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:47:11.982620 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:11.974584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35735 (local address 127.0.98.3:51897)
0620 14:47:11.974727 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:11.974761 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:11.975216 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:11.975474 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:11.975479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:11.976014 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:11.976201 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:11.976205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.976826 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.976827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:11.977606 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:11.977608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.977900 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.977901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:11.977993 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:11.978431 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:11.978447 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:11.979679 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:11.981256 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:11.981263 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:11.981265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:11.981542 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:11.981717 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:11.981718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:11.981718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:11.981820 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:11.982048 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:11.982054 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:11.982180 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 14:47:11.982265 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:47:11.982417 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260620 14:47:11.983031 11094 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35735
I20260620 14:47:11.983106 11094 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:11.983237 11094 heartbeater.cc:511] Master 127.0.98.62:35735 requested a full tablet report, sending...
I20260620 14:47:11.983546 10635 ts_manager.cc:194] Registered new tserver with Master: 03ba8a1beb2047e4bebca6d6e806fb5d (127.0.98.3:41781)
I20260620 14:47:11.983889 10635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51897
I20260620 14:47:11.992164 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:47:11.998850 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:11.993336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32912 (local address 127.0.98.62:35735)
0620 14:47:11.993385 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:11.993387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:11.993480 (+ 93us) server_negotiation.cc:407] Connection header received
0620 14:47:11.993545 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:11.993546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:11.993556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:11.993599 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:11.994104 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.994593 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.995299 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.995527 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.995580 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:11.996658 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:11.996660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:11.996660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:11.996668 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:11.997861 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:11.998152 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:11.998153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:11.998154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:11.998206 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:11.998359 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:11.998359 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:11.998360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:11.998505 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:11.998587 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:11.998693 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 14:47:11.998765 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:47:12.000267 10635 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:47:12.001760 10635 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:47:12.007972 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:12.003867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43786 (local address 127.0.98.1:36957)
0620 14:47:12.004391 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:12.004394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:12.004401 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:47:12.004438 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:12.004439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:12.004449 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:12.004511 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:12.004661 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.005459 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.006851 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.007593 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.007669 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:12.007734 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:12.007853 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 14:47:12.007893 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":92,"threads_started":1}
I20260620 14:47:12.007966 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:12.003763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43681 (local address 127.0.0.1:38738)
0620 14:47:12.004073 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:12.004081 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:12.004118 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:12.004294 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:12.004297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:12.004300 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:12.004359 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:12.004361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.005832 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.005833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:12.006864 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:12.006866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.007706 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.007707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:12.007776 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:12.007780 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:47:12.007813 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 14:47:12.007868 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260620 14:47:12.008198 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:12.003933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38738 (local address 127.0.98.2:43681)
0620 14:47:12.004120 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:12.004124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:12.004137 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:47:12.004177 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:12.004180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:12.004197 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:12.004274 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:12.004427 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.005722 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.006950 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.007619 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.008008 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:12.008074 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:12.008098 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 14:47:12.008139 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":71,"threads_started":1}
I20260620 14:47:12.007961 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:12.003763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36957 (local address 127.0.0.1:43786)
0620 14:47:12.003893 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:12.003935 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:12.004000 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:12.004530 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:12.004531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:12.004533 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:12.004586 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:12.004588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.005551 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.005552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:12.006780 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:12.006783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.007675 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.007676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:12.007752 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:12.007767 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 14:47:12.007832 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:47:12.007874 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":51,"threads_started":1}
I20260620 14:47:12.008793 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:12.003974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41781 (local address 127.0.0.1:43742)
0620 14:47:12.004161 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:12.004170 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:12.004217 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:12.004620 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:12.004621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:12.004622 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:12.004656 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:12.004658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.006957 (+ 2299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.006958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:12.007678 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:12.007679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.008568 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.008569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:12.008616 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:12.008620 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:47:12.008683 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:47:12.008726 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":62,"threads_started":1}
I20260620 14:47:12.009145 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:12.004171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43742 (local address 127.0.98.3:41781)
0620 14:47:12.004484 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:12.004488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:12.004498 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:47:12.004531 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:12.004533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:12.004547 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:12.004624 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:12.004698 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.005742 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.007764 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.008473 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.008548 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:12.008600 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:12.009018 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 14:47:12.009066 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":80,"threads_started":1}
I20260620 14:47:12.009071 10894 tablet_service.cc:1467] Processing CreateTablet for tablet ce65b803649749aa804dae8df6831df9 (DEFAULT_TABLE table=test-table [id=658a11cce0634e8995733e30e2dc63b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:12.009433 10894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce65b803649749aa804dae8df6831df9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:12.009737 11028 tablet_service.cc:1467] Processing CreateTablet for tablet ce65b803649749aa804dae8df6831df9 (DEFAULT_TABLE table=test-table [id=658a11cce0634e8995733e30e2dc63b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:12.010003 11028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce65b803649749aa804dae8df6831df9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:12.012070 11111 tablet_bootstrap.cc:492] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96: Bootstrap starting.
I20260620 14:47:12.012445 11112 tablet_bootstrap.cc:492] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d: Bootstrap starting.
I20260620 14:47:12.012615 10760 tablet_service.cc:1467] Processing CreateTablet for tablet ce65b803649749aa804dae8df6831df9 (DEFAULT_TABLE table=test-table [id=658a11cce0634e8995733e30e2dc63b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:12.012720 11111 tablet_bootstrap.cc:654] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:12.012936 10760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce65b803649749aa804dae8df6831df9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:12.013077 11112 tablet_bootstrap.cc:654] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:12.013227 11111 log.cc:824] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:12.013571 11112 log.cc:824] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:12.013998 11111 tablet_bootstrap.cc:492] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96: No bootstrap required, opened a new log
I20260620 14:47:12.014070 11111 ts_tablet_manager.cc:1397] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:47:12.014210 11112 tablet_bootstrap.cc:492] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d: No bootstrap required, opened a new log
I20260620 14:47:12.014267 11112 ts_tablet_manager.cc:1397] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:47:12.015724 11112 raft_consensus.cc:348] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049201653cde40d7bf36b114345edb96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43681 } } peers { permanent_uuid: "31e612e23a67415dad338f8f3d2197a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36957 } } peers { permanent_uuid: "03ba8a1beb2047e4bebca6d6e806fb5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41781 } }
I20260620 14:47:12.015856 11112 raft_consensus.cc:374] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:12.015887 11112 raft_consensus.cc:729] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ba8a1beb2047e4bebca6d6e806fb5d, State: Initialized, Role: FOLLOWER
I20260620 14:47:12.016012 11112 consensus_queue.cc:260] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049201653cde40d7bf36b114345edb96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43681 } } peers { permanent_uuid: "31e612e23a67415dad338f8f3d2197a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36957 } } peers { permanent_uuid: "03ba8a1beb2047e4bebca6d6e806fb5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41781 } }
I20260620 14:47:12.016065 11111 raft_consensus.cc:348] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049201653cde40d7bf36b114345edb96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43681 } } peers { permanent_uuid: "31e612e23a67415dad338f8f3d2197a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36957 } } peers { permanent_uuid: "03ba8a1beb2047e4bebca6d6e806fb5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41781 } }
I20260620 14:47:12.016244 11111 raft_consensus.cc:374] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:12.016266 11112 ts_tablet_manager.cc:1428] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:47:12.016292 11111 raft_consensus.cc:729] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 049201653cde40d7bf36b114345edb96, State: Initialized, Role: FOLLOWER
I20260620 14:47:12.016383 11094 heartbeater.cc:503] Master 127.0.98.62:35735 was elected leader, sending a full tablet report...
I20260620 14:47:12.016400 11111 consensus_queue.cc:260] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049201653cde40d7bf36b114345edb96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43681 } } peers { permanent_uuid: "31e612e23a67415dad338f8f3d2197a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36957 } } peers { permanent_uuid: "03ba8a1beb2047e4bebca6d6e806fb5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41781 } }
I20260620 14:47:12.016631 11115 tablet_bootstrap.cc:492] T ce65b803649749aa804dae8df6831df9 P 31e612e23a67415dad338f8f3d2197a8: Bootstrap starting.
I20260620 14:47:12.016692 10960 heartbeater.cc:503] Master 127.0.98.62:35735 was elected leader, sending a full tablet report...
I20260620 14:47:12.016687 11111 ts_tablet_manager.cc:1428] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:47:12.017279 11115 tablet_bootstrap.cc:654] T ce65b803649749aa804dae8df6831df9 P 31e612e23a67415dad338f8f3d2197a8: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:12.017766 11115 log.cc:824] T ce65b803649749aa804dae8df6831df9 P 31e612e23a67415dad338f8f3d2197a8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:12.018301 11115 tablet_bootstrap.cc:492] T ce65b803649749aa804dae8df6831df9 P 31e612e23a67415dad338f8f3d2197a8: No bootstrap required, opened a new log
I20260620 14:47:12.018349 11115 ts_tablet_manager.cc:1397] T ce65b803649749aa804dae8df6831df9 P 31e612e23a67415dad338f8f3d2197a8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:47:12.019712 11115 raft_consensus.cc:348] T ce65b803649749aa804dae8df6831df9 P 31e612e23a67415dad338f8f3d2197a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049201653cde40d7bf36b114345edb96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43681 } } peers { permanent_uuid: "31e612e23a67415dad338f8f3d2197a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36957 } } peers { permanent_uuid: "03ba8a1beb2047e4bebca6d6e806fb5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41781 } }
I20260620 14:47:12.019836 11115 raft_consensus.cc:374] T ce65b803649749aa804dae8df6831df9 P 31e612e23a67415dad338f8f3d2197a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:12.019855 11115 raft_consensus.cc:729] T ce65b803649749aa804dae8df6831df9 P 31e612e23a67415dad338f8f3d2197a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31e612e23a67415dad338f8f3d2197a8, State: Initialized, Role: FOLLOWER
I20260620 14:47:12.019943 11115 consensus_queue.cc:260] T ce65b803649749aa804dae8df6831df9 P 31e612e23a67415dad338f8f3d2197a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049201653cde40d7bf36b114345edb96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43681 } } peers { permanent_uuid: "31e612e23a67415dad338f8f3d2197a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36957 } } peers { permanent_uuid: "03ba8a1beb2047e4bebca6d6e806fb5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41781 } }
I20260620 14:47:12.020148 11115 ts_tablet_manager.cc:1428] T ce65b803649749aa804dae8df6831df9 P 31e612e23a67415dad338f8f3d2197a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:47:12.020203 10826 heartbeater.cc:503] Master 127.0.98.62:35735 was elected leader, sending a full tablet report...
I20260620 14:47:12.069607 11116 raft_consensus.cc:482] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:47:12.069726 11116 raft_consensus.cc:504] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049201653cde40d7bf36b114345edb96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43681 } } peers { permanent_uuid: "31e612e23a67415dad338f8f3d2197a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36957 } } peers { permanent_uuid: "03ba8a1beb2047e4bebca6d6e806fb5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41781 } }
I20260620 14:47:12.070029 11116 leader_election.cc:288] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 049201653cde40d7bf36b114345edb96 (127.0.98.2:43681), 31e612e23a67415dad338f8f3d2197a8 (127.0.98.1:36957)
I20260620 14:47:12.072829 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:12.070307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45403 (local address 127.0.98.2:43681)
0620 14:47:12.070351 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:12.070354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:12.070361 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:47:12.070386 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:12.070387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:12.070398 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:12.070455 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:12.070621 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.071133 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.072144 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.072502 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.072557 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:12.072568 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:12.072731 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:47:12.072768 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:47:12.072830 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:12.070327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39543 (local address 127.0.98.1:36957)
0620 14:47:12.070376 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:12.070377 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:12.070472 (+ 95us) server_negotiation.cc:407] Connection header received
0620 14:47:12.070508 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:12.070509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:12.070522 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:12.070596 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:12.070724 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.071186 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.072138 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.072511 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.072581 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:12.072592 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:12.072707 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 14:47:12.072746 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:47:12.072836 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:12.070216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36957 (local address 127.0.98.3:39543)
0620 14:47:12.070412 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:12.070420 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:12.070466 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:12.070598 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:12.070598 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:12.070600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:12.070666 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:12.070667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.071263 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.071263 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:12.072051 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:12.072052 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.072590 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.072591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:12.072644 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:12.072648 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:47:12.072687 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 14:47:12.072722 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":124,"threads_started":1}
I20260620 14:47:12.072841 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:12.070191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43681 (local address 127.0.98.3:45403)
0620 14:47:12.070242 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:12.070255 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:12.070299 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:12.070475 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:12.070477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:12.070479 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:12.070552 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:12.070554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.071208 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.071209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:12.072050 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:12.072053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.072577 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.072578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:12.072659 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:12.072662 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:47:12.072720 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:47:12.072758 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":19}
I20260620 14:47:12.073223 10780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce65b803649749aa804dae8df6831df9" candidate_uuid: "03ba8a1beb2047e4bebca6d6e806fb5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31e612e23a67415dad338f8f3d2197a8" is_pre_election: true
I20260620 14:47:12.073223 10914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce65b803649749aa804dae8df6831df9" candidate_uuid: "03ba8a1beb2047e4bebca6d6e806fb5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049201653cde40d7bf36b114345edb96" is_pre_election: true
I20260620 14:47:12.073386 10914 raft_consensus.cc:2454] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03ba8a1beb2047e4bebca6d6e806fb5d in term 0.
I20260620 14:47:12.073386 10780 raft_consensus.cc:2454] T ce65b803649749aa804dae8df6831df9 P 31e612e23a67415dad338f8f3d2197a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03ba8a1beb2047e4bebca6d6e806fb5d in term 0.
I20260620 14:47:12.073614 10984 leader_election.cc:302] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ba8a1beb2047e4bebca6d6e806fb5d, 31e612e23a67415dad338f8f3d2197a8; no voters:
I20260620 14:47:12.073736 11116 raft_consensus.cc:2781] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:47:12.073796 11116 raft_consensus.cc:482] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:47:12.073817 11116 raft_consensus.cc:3037] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:12.074584 11116 raft_consensus.cc:504] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049201653cde40d7bf36b114345edb96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43681 } } peers { permanent_uuid: "31e612e23a67415dad338f8f3d2197a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36957 } } peers { permanent_uuid: "03ba8a1beb2047e4bebca6d6e806fb5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41781 } }
I20260620 14:47:12.074702 11116 leader_election.cc:288] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [CANDIDATE]: Term 1 election: Requested vote from peers 049201653cde40d7bf36b114345edb96 (127.0.98.2:43681), 31e612e23a67415dad338f8f3d2197a8 (127.0.98.1:36957)
I20260620 14:47:12.074857 10780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce65b803649749aa804dae8df6831df9" candidate_uuid: "03ba8a1beb2047e4bebca6d6e806fb5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31e612e23a67415dad338f8f3d2197a8"
I20260620 14:47:12.074884 10914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce65b803649749aa804dae8df6831df9" candidate_uuid: "03ba8a1beb2047e4bebca6d6e806fb5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049201653cde40d7bf36b114345edb96"
I20260620 14:47:12.074942 10914 raft_consensus.cc:3037] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:12.074940 10780 raft_consensus.cc:3037] T ce65b803649749aa804dae8df6831df9 P 31e612e23a67415dad338f8f3d2197a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:12.075613 10914 raft_consensus.cc:2454] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ba8a1beb2047e4bebca6d6e806fb5d in term 1.
I20260620 14:47:12.075687 10780 raft_consensus.cc:2454] T ce65b803649749aa804dae8df6831df9 P 31e612e23a67415dad338f8f3d2197a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ba8a1beb2047e4bebca6d6e806fb5d in term 1.
I20260620 14:47:12.075773 10983 leader_election.cc:302] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ba8a1beb2047e4bebca6d6e806fb5d, 049201653cde40d7bf36b114345edb96; no voters:
I20260620 14:47:12.075865 11116 raft_consensus.cc:2781] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:12.076020 11116 raft_consensus.cc:686] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [term 1 LEADER]: Becoming Leader. State: Replica: 03ba8a1beb2047e4bebca6d6e806fb5d, State: Running, Role: LEADER
I20260620 14:47:12.076109 11116 consensus_queue.cc:237] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049201653cde40d7bf36b114345edb96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43681 } } peers { permanent_uuid: "31e612e23a67415dad338f8f3d2197a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36957 } } peers { permanent_uuid: "03ba8a1beb2047e4bebca6d6e806fb5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41781 } }
I20260620 14:47:12.076900 10635 catalog_manager.cc:5495] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 03ba8a1beb2047e4bebca6d6e806fb5d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "03ba8a1beb2047e4bebca6d6e806fb5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049201653cde40d7bf36b114345edb96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31e612e23a67415dad338f8f3d2197a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03ba8a1beb2047e4bebca6d6e806fb5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41781 } health_report { overall_health: HEALTHY } } }
W20260620 14:47:12.081669 10961 tablet.cc:2369] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:47:12.098627 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:12.096454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43744 (local address 127.0.98.3:41781)
0620 14:47:12.096522 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:12.096524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:12.096546 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:47:12.096582 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:12.096582 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:12.096600 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:12.096658 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:12.096770 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.097226 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.097956 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.098140 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.098192 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:12.098328 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:12.098425 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:12.098503 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:12.098519 (+ 16us) server_negotiation.cc:299] Negotiation successful
0620 14:47:12.098538 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:47:12.100299 10914 raft_consensus.cc:1261] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96 [term 1 FOLLOWER]: Refusing update from remote peer 03ba8a1beb2047e4bebca6d6e806fb5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:12.100301 10780 raft_consensus.cc:1261] T ce65b803649749aa804dae8df6831df9 P 31e612e23a67415dad338f8f3d2197a8 [term 1 FOLLOWER]: Refusing update from remote peer 03ba8a1beb2047e4bebca6d6e806fb5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:12.100634 11116 consensus_queue.cc:1035] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "31e612e23a67415dad338f8f3d2197a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:12.100773 11121 consensus_queue.cc:1035] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "049201653cde40d7bf36b114345edb96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:12.103230 11125 mvcc.cc:205] Tried to move back new op lower bound from 7298936144280440832 to 7298936144184471552. Current Snapshot: MvccSnapshot[applied={T|T < 7298936144280440832}]
I20260620 14:47:12.105223 10634 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:12.105310 10634 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:12.105962 10634 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 966becd32e9447d0819095951f49084c: Sending DeleteTablet for 3 replicas of tablet ce65b803649749aa804dae8df6831df9
I20260620 14:47:12.106242 10894 tablet_service.cc:1514] Processing DeleteTablet for tablet ce65b803649749aa804dae8df6831df9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38738
I20260620 14:47:12.106302 10760 tablet_service.cc:1514] Processing DeleteTablet for tablet ce65b803649749aa804dae8df6831df9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43786
I20260620 14:47:12.106334 11028 tablet_service.cc:1514] Processing DeleteTablet for tablet ce65b803649749aa804dae8df6831df9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43742
I20260620 14:47:12.106621 11133 tablet_replica.cc:330] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96: stopping tablet replica
I20260620 14:47:12.106650 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 10697
I20260620 14:47:12.106727 11133 raft_consensus.cc:2229] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:12.106828 11133 raft_consensus.cc:2258] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:12.108078 11133 ts_tablet_manager.cc:1905] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:12.109798 11133 ts_tablet_manager.cc:1918] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:47:12.109874 11133 log.cc:1196] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/ce65b803649749aa804dae8df6831df9
I20260620 14:47:12.110131 11133 ts_tablet_manager.cc:1939] T ce65b803649749aa804dae8df6831df9 P 049201653cde40d7bf36b114345edb96: Deleting consensus metadata
I20260620 14:47:12.110508 10621 catalog_manager.cc:4841] TS 049201653cde40d7bf36b114345edb96 (127.0.98.2:43681): tablet ce65b803649749aa804dae8df6831df9 (table test-table [id=658a11cce0634e8995733e30e2dc63b3]) successfully deleted
W20260620 14:47:12.113899 10622 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36957 (error 108)
I20260620 14:47:12.113997 11135 tablet_replica.cc:330] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d: stopping tablet replica
I20260620 14:47:12.114104 11135 raft_consensus.cc:2229] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:47:12.114239 11135 raft_consensus.cc:2258] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:12.114460 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:12.114321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36957 (local address 127.0.0.1:43800)
0620 14:47:12.114391 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:12.114425 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 14:47:12.114516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 10831
W20260620 14:47:12.114663 10622 catalog_manager.cc:4584] TS 31e612e23a67415dad338f8f3d2197a8 (127.0.98.1:36957): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce65b803649749aa804dae8df6831df9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36957: connect: Connection refused (error 111)
I20260620 14:47:12.115170 11135 ts_tablet_manager.cc:1905] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:12.116688 11135 ts_tablet_manager.cc:1918] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:47:12.116758 11135 log.cc:1196] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/ce65b803649749aa804dae8df6831df9
I20260620 14:47:12.117007 11135 ts_tablet_manager.cc:1939] T ce65b803649749aa804dae8df6831df9 P 03ba8a1beb2047e4bebca6d6e806fb5d: Deleting consensus metadata
I20260620 14:47:12.118520 10620 catalog_manager.cc:4841] TS 03ba8a1beb2047e4bebca6d6e806fb5d (127.0.98.3:41781): tablet ce65b803649749aa804dae8df6831df9 (table test-table [id=658a11cce0634e8995733e30e2dc63b3]) successfully deleted
I20260620 14:47:12.119263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 10965
I20260620 14:47:12.123055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 10603
2026-06-20T14:47:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1846 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:47:12 dist-test-slave-cx89 krb5kdc[11139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:47:12 dist-test-slave-cx89 krb5kdc[11139](info): set up 2 sockets
Jun 20 14:47:12 dist-test-slave-cx89 krb5kdc[11139](info): commencing operation
krb5kdc: starting...
W20260620 14:47:13.071678 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:47:13 dist-test-slave-cx89 krb5kdc[11139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:47:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 14:47:13.215476 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskV7d6BE/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/postgres -l logfile start
2026-06-20 14:47:15.428 UTC [11169] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:47:15.428 UTC [11169] LOG: listening on IPv4 address "127.0.98.20", port 56069
2026-06-20 14:47:15.431 UTC [11169] LOG: listening on Unix socket "/tmp/.s.PGSQL.56069"
2026-06-20 14:47:15.451 UTC [11172] LOG: database system was shut down at 2026-06-20 14:47:14 UTC
2026-06-20 14:47:15.454 UTC [11169] LOG: database system is ready to accept connections
I20260620 14:47:16.331164 392 mini_postgres.cc:96] Postgres bound to 56069
2026-06-20 14:47:16.336 UTC [11180] FATAL: database "slave" does not exist
127.0.98.20:56069 - accepting connections
I20260620 14:47:16.337236 392 mini_ranger.cc:161] Starting Ranger...
I20260620 14:47:16.350951 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:47:16.786808 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:47:16.786883 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ranger-admin
2026-06-20 14:47:17,048 [I] DB FLAVOR :POSTGRES
2026-06-20 14:47:17,049 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:47:17,049 [I] Checking connection..
2026-06-20 14:47:17,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:47:17,276 [I] Checking connection passed.
2026-06-20 14:47:17,276 [I] --------- Verifying version history table ---------
2026-06-20 14:47:17,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:47:17,500 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:47:17,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:47:17,751 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:47:17,751 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:47:17,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:47:17.976 UTC [11244] WARNING: there is no transaction in progress
2026-06-20 14:47:17,982 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:47:17,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:47:18,236 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:47:18,237 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:47:18,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:47:18,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:47:18,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cx89.c.gcp-upstream.internal','N') ;"
2026-06-20 14:47:18,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:47:19,188 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:47:19,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:47:19,437 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:47:19,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:47:19,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:47:19,896 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:47:19,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:47:20.788 UTC [11352] WARNING: there is no transaction in progress
2026-06-20 14:47:20.804 UTC [11352] WARNING: there is no transaction in progress
2026-06-20 14:47:20.819 UTC [11352] WARNING: there is no transaction in progress
2026-06-20 14:47:20.833 UTC [11352] WARNING: there is no transaction in progress
2026-06-20 14:47:20.851 UTC [11352] WARNING: there is no transaction in progress
2026-06-20 14:47:20.871 UTC [11352] WARNING: there is no transaction in progress
2026-06-20 14:47:20.927 UTC [11352] WARNING: there is no transaction in progress
2026-06-20 14:47:20.933 UTC [11352] WARNING: there is no transaction in progress
2026-06-20 14:47:20.947 UTC [11352] WARNING: there is no transaction in progress
2026-06-20 14:47:20.957 UTC [11352] WARNING: there is no transaction in progress
2026-06-20 14:47:20.967 UTC [11352] WARNING: there is no transaction in progress
2026-06-20 14:47:20.980 UTC [11352] WARNING: there is no transaction in progress
2026-06-20 14:47:20.989 UTC [11352] WARNING: there is no transaction in progress
2026-06-20 14:47:20.994 UTC [11352] WARNING: there is no transaction in progress
2026-06-20 14:47:21.001 UTC [11352] WARNING: there is no transaction in progress
2026-06-20 14:47:21.007 UTC [11352] WARNING: there is no transaction in progress
2026-06-20 14:47:21,591 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:47:21,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:47:21,830 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:47:21,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:47:22,064 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:47:22,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:47:22,315 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:47:22,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:47:22,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:47:22,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cx89.c.gcp-upstream.internal';"
2026-06-20 14:47:23,044 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:47:23,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:47:23,292 [I] DB_PATCHES have already been applied
I20260620 14:47:23.297657 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskV7d6BE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:47:23.297724 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 14:47:23.300706 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:44701
Jun 20, 2026 2:47:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:47:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:47:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:47:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:47:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:47:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:47:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:47:24.308243 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.971s user 0.000s sys 0.008s
Jun 20, 2026 2:47:43 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:47:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 2:47:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:47:43 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:47:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:47:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:47:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:47:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:47:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:47:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:47:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:47:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:47:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:47:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:47:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:47:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:47:43 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:47:44 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:47:44.622491 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:47:44 dist-test-slave-cx89 krb5kdc[11139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966864, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:47:44.696458 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:47:44.705245 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:47:45.474365 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:47:45.474440 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ranger-kms
I20260620 14:47:45.474457 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:56069
2026-06-20 14:47:45,632 [I] DB FLAVOR :POSTGRES
2026-06-20 14:47:45,633 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:47:45,633 [I] Checking connection
2026-06-20 14:47:45,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:47:45,888 [I] connection success
2026-06-20 14:47:45,888 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:47:45,888 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:47:45,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:47:46,110 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:47:46,110 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:47:46,111 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:47:46,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56069/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:47:46,379 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:47:46.831935 392 mini_ranger_kms.cc:310] Created kms service
I20260620 14:47:47.013942 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:47:47.052919 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:47:47 dist-test-slave-cx89 krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966864, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:47:47.403555 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:47:47.403666 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:47:47.403685 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 14:47:47.406955 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11532
I20260620 14:47:47.407078 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:47569
Jun 20, 2026 2:47:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:47:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:47:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:47:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:47:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:47:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 14:47:49.363128 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.667s user 0.017s sys 0.124s
I20260620 14:47:49.363229 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:47:49.363296 392 mini_ranger_kms.cc:401] 127.0.98.20:47569/kms/v1/keys
Jun 20 14:47:49 dist-test-slave-cx89 krb5kdc[11139](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781966869, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 14:47:49 dist-test-slave-cx89 krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966869, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:47:50 dist-test-slave-cx89 krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966869, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:47:52 dist-test-slave-cx89 krb5kdc[11139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:47:52.374285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39689
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39689
--ranger_config_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47569/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:52.474987 11583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:52.475219 11583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:52.475250 11583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:52.476454 11583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:52.476497 11583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:52.476516 11583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:52.476532 11583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:52.476548 11583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:47:52.478097 11583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:47569/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39689
--ranger_config_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:47:52.478334 11583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:52.478531 11583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:52.483096 11588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:52.483181 11589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:52.483096 11591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:52.483743 11583 server_base.cc:1034] running on GCE node
I20260620 14:47:52.484004 11583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:52.484359 11583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:52.485534 11583 hybrid_clock.cc:648] HybridClock initialized: now 1781966872485515 us; error 34 us; skew 500 ppm
Jun 20 14:47:52 dist-test-slave-cx89 krb5kdc[11139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:52.488447 11583 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:47:52.489003 11583 webserver.cc:460] Webserver started at http://127.0.98.62:35307/ using document root <none> and password file <none>
I20260620 14:47:52.489234 11583 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:52.489287 11583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:52.489418 11583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:47:52 dist-test-slave-cx89 krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:47:52.550042 11583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "0baf7ae3146c4bd0a217e36f0c7d8ad2"
format_stamp: "Formatted at 2026-06-20 14:47:52 on dist-test-slave-cx89"
server_key: "6fac93da44efc745d713c9f232cafe98"
server_key_iv: "938c2809a56ad6099263d12afe29adb0"
server_key_version: "kuduclusterkey@0"
I20260620 14:47:52.550442 11583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "0baf7ae3146c4bd0a217e36f0c7d8ad2"
format_stamp: "Formatted at 2026-06-20 14:47:52 on dist-test-slave-cx89"
server_key: "6fac93da44efc745d713c9f232cafe98"
server_key_iv: "938c2809a56ad6099263d12afe29adb0"
server_key_version: "kuduclusterkey@0"
I20260620 14:47:52.551813 11583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:47:52.566165 11598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:52.566375 11583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:47:52.566485 11583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "0baf7ae3146c4bd0a217e36f0c7d8ad2"
format_stamp: "Formatted at 2026-06-20 14:47:52 on dist-test-slave-cx89"
server_key: "6fac93da44efc745d713c9f232cafe98"
server_key_iv: "938c2809a56ad6099263d12afe29adb0"
server_key_version: "kuduclusterkey@0"
I20260620 14:47:52.566581 11583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:52.602625 11583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:52.604737 11583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:52.604885 11583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:52.610096 11583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39689
I20260620 14:47:52.610128 11650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39689 every 8 connection(s)
I20260620 14:47:52.610541 11583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:52.611182 11651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:52.613395 11651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2: Bootstrap starting.
I20260620 14:47:52.613936 11651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:52.614166 11651 log.cc:824] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:52.614778 11651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2: No bootstrap required, opened a new log
I20260620 14:47:52.617172 11651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0baf7ae3146c4bd0a217e36f0c7d8ad2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39689 } }
I20260620 14:47:52.617316 11651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:52.617347 11651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0baf7ae3146c4bd0a217e36f0c7d8ad2, State: Initialized, Role: FOLLOWER
I20260620 14:47:52.617481 11651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0baf7ae3146c4bd0a217e36f0c7d8ad2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39689 } }
I20260620 14:47:52.617554 11651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:52.617583 11651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:52.617630 11651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:52.618134 11651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0baf7ae3146c4bd0a217e36f0c7d8ad2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39689 } }
I20260620 14:47:52.618223 11651 leader_election.cc:302] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0baf7ae3146c4bd0a217e36f0c7d8ad2; no voters:
I20260620 14:47:52.618372 11651 leader_election.cc:288] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:52.618415 11654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:52.618602 11651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:52.618615 11654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2 [term 1 LEADER]: Becoming Leader. State: Replica: 0baf7ae3146c4bd0a217e36f0c7d8ad2, State: Running, Role: LEADER
I20260620 14:47:52.618784 11654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0baf7ae3146c4bd0a217e36f0c7d8ad2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39689 } }
I20260620 14:47:52.619153 11655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0baf7ae3146c4bd0a217e36f0c7d8ad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0baf7ae3146c4bd0a217e36f0c7d8ad2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39689 } } }
I20260620 14:47:52.619138 11656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0baf7ae3146c4bd0a217e36f0c7d8ad2. Latest consensus state: current_term: 1 leader_uuid: "0baf7ae3146c4bd0a217e36f0c7d8ad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0baf7ae3146c4bd0a217e36f0c7d8ad2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39689 } } }
I20260620 14:47:52.619270 11655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:52.619278 11656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:52.619481 11663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:52.619930 11663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:52.620143 11651 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:47:52.620779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 11583
I20260620 14:47:52.620869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:47:52.621034 392 external_mini_cluster.cc:1428] Setting key 4586b9f06ec5ed6ffd39e3d818e0d4b2
I20260620 14:47:52.628728 11673 mvcc.cc:205] Tried to move back new op lower bound from 7298936310266687488 to 7298936310246883328. Current Snapshot: MvccSnapshot[applied={T|T < 7298936310266687488}]
I20260620 14:47:52.629669 11663 catalog_manager.cc:1347] Generated new cluster ID: 88622642437c40d6a25e8f9cde3587c8
I20260620 14:47:52.629746 11663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:47:52 dist-test-slave-cx89 krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:47:52.636471 11674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:52.622961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.0.98.62:39689)
0620 14:47:52.624054 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:52.624065 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:52.624106 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:47:52.624289 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:52.624295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:52.624338 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:52.624475 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:52.625146 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:52.626559 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:52.627304 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:52.628031 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:52.628162 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:52.632800 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:52.632803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:52.632804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:52.632815 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:52.634793 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:52.635144 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:52.635147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:52.635147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:52.635232 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:52.635404 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:52.635405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:52.635405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:52.635759 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:52.635861 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:52.636097 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 14:47:52.636229 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1017,"thread_start_us":83,"threads_started":1}
I20260620 14:47:52.637303 11663 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:52.638484 11663 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:52.643585 11663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2: Generated new TSK 0
I20260620 14:47:52.643965 11663 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:47:53.390560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47569/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39689
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:53.494590 11696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:53.494872 11696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:53.494936 11696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:53.496155 11696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:53.496193 11696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:53.496230 11696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:47:53.497684 11696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:47569/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:47:53.497920 11696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:53.498118 11696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:53.504154 11702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:53.504154 11701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:53.504246 11696 server_base.cc:1034] running on GCE node
W20260620 14:47:53.504407 11704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:53.504626 11696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:53.504891 11696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:53.506052 11696 hybrid_clock.cc:648] HybridClock initialized: now 1781966873506035 us; error 32 us; skew 500 ppm
Jun 20 14:47:53 dist-test-slave-cx89 krb5kdc[11139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:53.508919 11696 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:47:53.509470 11696 webserver.cc:460] Webserver started at http://127.0.98.1:33389/ using document root <none> and password file <none>
I20260620 14:47:53.509675 11696 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:53.509717 11696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:53.509825 11696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:47:53 dist-test-slave-cx89 krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:47:53.524366 11696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "1bca1be0032c484eb868b7ba055fdfba"
format_stamp: "Formatted at 2026-06-20 14:47:53 on dist-test-slave-cx89"
server_key: "b16c6f5eee0c90109e3face409e9a971"
server_key_iv: "2ae9e51b13ae555d5220f42d8942300b"
server_key_version: "kuduclusterkey@0"
I20260620 14:47:53.524931 11696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1bca1be0032c484eb868b7ba055fdfba"
format_stamp: "Formatted at 2026-06-20 14:47:53 on dist-test-slave-cx89"
server_key: "b16c6f5eee0c90109e3face409e9a971"
server_key_iv: "2ae9e51b13ae555d5220f42d8942300b"
server_key_version: "kuduclusterkey@0"
I20260620 14:47:53.526722 11696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:47:53.549300 11713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:53.549574 11696 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260620 14:47:53.549682 11696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "1bca1be0032c484eb868b7ba055fdfba"
format_stamp: "Formatted at 2026-06-20 14:47:53 on dist-test-slave-cx89"
server_key: "b16c6f5eee0c90109e3face409e9a971"
server_key_iv: "2ae9e51b13ae555d5220f42d8942300b"
server_key_version: "kuduclusterkey@0"
I20260620 14:47:53.549808 11696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:53.585402 11696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:53.587895 11696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:53.588110 11696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:53.588402 11696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:53.588806 11696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:53.588897 11696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:53.588979 11696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:53.589043 11696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:53.596167 11696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33447
I20260620 14:47:53.596674 11696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:47:53.598143 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 11696
I20260620 14:47:53.598244 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:47:53.598265 11826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33447 every 8 connection(s)
I20260620 14:47:53.598430 392 external_mini_cluster.cc:1428] Setting key 9b464574c426ba3ab41586ce23c3835b
Jun 20 14:47:53 dist-test-slave-cx89 krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:47:53.609999 11829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:53.599096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39689 (local address 127.0.98.1:36719)
0620 14:47:53.599628 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:53.599673 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:53.600282 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:53.600619 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:53.600623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:53.600862 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:53.601143 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:53.601149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.602175 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.602176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:53.603432 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:53.603435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.603956 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.603957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:53.604101 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:53.604683 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:53.604693 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:53.606370 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:53.608355 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:53.608363 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:53.608365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:53.608639 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:53.608862 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:53.608863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:53.608864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:53.608994 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:53.609288 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:53.609295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:53.609472 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 14:47:53.609587 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:47:53.609745 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":84,"threads_started":1}
I20260620 14:47:53.610538 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:53.599371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36719 (local address 127.0.98.62:39689)
0620 14:47:53.599571 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:53.599573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:53.600417 (+ 844us) server_negotiation.cc:407] Connection header received
0620 14:47:53.600443 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:53.600444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:53.600457 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:53.600508 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:53.601251 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.602074 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.603522 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.603818 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.603885 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:53.606478 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:53.606480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:53.606480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:53.606488 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:53.608266 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:53.608716 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:53.608717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:53.608717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:53.608793 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:53.609060 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:53.609061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:53.609062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:53.609200 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:53.609261 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:53.610361 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0620 14:47:53.610447 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":115,"threads_started":1}
I20260620 14:47:53.613910 11827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39689
I20260620 14:47:53.614006 11827 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:53.614202 11827 heartbeater.cc:511] Master 127.0.98.62:39689 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:47:53.614763 11615 ts_manager.cc:194] Registered new tserver with Master: 1bca1be0032c484eb868b7ba055fdfba (127.0.98.1:33447)
I20260620 14:47:53.615568 11615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:47:53.638007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47569/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39689
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:47:53 dist-test-slave-cx89 krb5kdc[11139](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781966873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 14:47:53.724530 11835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:53.724920 11835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:53.725008 11835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:53.726819 11835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:53.726941 11835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:53.727034 11835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:47:53.729209 11835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:47569/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:47:53.729662 11835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:53.729916 11835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:53.735359 11843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:53.735697 11841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:53.735939 11840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:53.737083 11835 server_base.cc:1034] running on GCE node
I20260620 14:47:53.737391 11835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:53.737664 11835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:53.738819 11835 hybrid_clock.cc:648] HybridClock initialized: now 1781966873738807 us; error 44 us; skew 500 ppm
Jun 20 14:47:53 dist-test-slave-cx89 krb5kdc[11139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:53.741236 11835 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:47:53.741748 11835 webserver.cc:460] Webserver started at http://127.0.98.2:35789/ using document root <none> and password file <none>
I20260620 14:47:53.742024 11835 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:53.742079 11835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:53.742198 11835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:47:53 dist-test-slave-cx89 krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:47:53.756690 11835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "03eb284351264d88a4aa5016829a9f5a"
format_stamp: "Formatted at 2026-06-20 14:47:53 on dist-test-slave-cx89"
server_key: "b1d3598d60f07a21090e6804f8bca49c"
server_key_iv: "a92b1b0d9989614fa02b33c8f5620883"
server_key_version: "kuduclusterkey@0"
I20260620 14:47:53.757143 11835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "03eb284351264d88a4aa5016829a9f5a"
format_stamp: "Formatted at 2026-06-20 14:47:53 on dist-test-slave-cx89"
server_key: "b1d3598d60f07a21090e6804f8bca49c"
server_key_iv: "a92b1b0d9989614fa02b33c8f5620883"
server_key_version: "kuduclusterkey@0"
I20260620 14:47:53.758888 11835 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:47:53.772301 11850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:53.772601 11835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:47:53.772691 11835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "03eb284351264d88a4aa5016829a9f5a"
format_stamp: "Formatted at 2026-06-20 14:47:53 on dist-test-slave-cx89"
server_key: "b1d3598d60f07a21090e6804f8bca49c"
server_key_iv: "a92b1b0d9989614fa02b33c8f5620883"
server_key_version: "kuduclusterkey@0"
I20260620 14:47:53.772786 11835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:53.800956 11835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:53.804209 11835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:53.804409 11835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:53.804693 11835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:53.805099 11835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:53.805150 11835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:53.805181 11835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:53.805244 11835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:53.816946 11835 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41223
I20260620 14:47:53.817035 11964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41223 every 8 connection(s)
I20260620 14:47:53.817459 11835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:47:53.823354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 11835
I20260620 14:47:53.823477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:47:53.823758 392 external_mini_cluster.cc:1428] Setting key 9bf973a74ada500b2324422ed2968eb6
Jun 20 14:47:53 dist-test-slave-cx89 krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:47:53.833150 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:53.820679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54321 (local address 127.0.98.62:39689)
0620 14:47:53.820760 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:53.820762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:53.820771 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:47:53.820799 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:53.820800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:53.820810 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:53.820871 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:53.824410 (+ 3539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.825124 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.826600 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.826966 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.827019 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:53.829719 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:53.829720 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:53.829721 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:53.829728 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:53.831328 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:53.831876 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:53.831878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:53.831878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:53.831943 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:53.832240 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:53.832241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:53.832242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:53.832371 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:53.832427 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:53.832963 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 14:47:53.833053 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:47:53.833312 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:53.818444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39689 (local address 127.0.98.2:54321)
0620 14:47:53.818606 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:53.818653 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:53.819210 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:53.823572 (+ 4362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:53.823577 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:53.823813 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:53.824102 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:53.824108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.825224 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.825225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:53.826500 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:53.826503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.827115 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.827116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:53.827270 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:53.828329 (+ 1059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:53.828339 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:53.829630 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:53.831408 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:53.831436 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:53.831439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:53.831727 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:53.832016 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:53.832017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:53.832018 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:53.832154 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:53.832553 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:53.832560 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:53.832741 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 14:47:53.832905 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 14:47:53.833072 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260620 14:47:53.833853 11965 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39689
I20260620 14:47:53.833947 11965 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:53.834131 11965 heartbeater.cc:511] Master 127.0.98.62:39689 requested a full tablet report, sending...
I20260620 14:47:53.834561 11615 ts_manager.cc:194] Registered new tserver with Master: 03eb284351264d88a4aa5016829a9f5a (127.0.98.2:41223)
I20260620 14:47:53.835052 11615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54321
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:47:53.878342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47569/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39689
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:53.990226 11970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:53.990618 11970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:53.990676 11970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:53.992607 11970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:53.992746 11970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:53.992861 11970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:47:53.995234 11970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:47569/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:47:53.995741 11970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:53.996068 11970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:54.004282 11976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:54.004674 11975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:54.004588 11978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:54.004789 11970 server_base.cc:1034] running on GCE node
I20260620 14:47:54.005117 11970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:54.005419 11970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:54.006610 11970 hybrid_clock.cc:648] HybridClock initialized: now 1781966874006590 us; error 34 us; skew 500 ppm
Jun 20 14:47:54 dist-test-slave-cx89 krb5kdc[11139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:54.010910 11970 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:47:54.011514 11970 webserver.cc:460] Webserver started at http://127.0.98.3:40525/ using document root <none> and password file <none>
I20260620 14:47:54.011790 11970 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:54.011870 11970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:54.012019 11970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:47:54 dist-test-slave-cx89 krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:47:54.029616 11970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "38b4b4b68b6149cd82a77dc6814f70da"
format_stamp: "Formatted at 2026-06-20 14:47:54 on dist-test-slave-cx89"
server_key: "81cdd2bdab84abaf230c34f8d38b30d5"
server_key_iv: "733c61f15731d91ed4f2df80edf94d42"
server_key_version: "kuduclusterkey@0"
I20260620 14:47:54.030135 11970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "38b4b4b68b6149cd82a77dc6814f70da"
format_stamp: "Formatted at 2026-06-20 14:47:54 on dist-test-slave-cx89"
server_key: "81cdd2bdab84abaf230c34f8d38b30d5"
server_key_iv: "733c61f15731d91ed4f2df80edf94d42"
server_key_version: "kuduclusterkey@0"
I20260620 14:47:54.035135 11970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.001s
I20260620 14:47:54.049386 11985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:54.049616 11970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 14:47:54.049718 11970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "38b4b4b68b6149cd82a77dc6814f70da"
format_stamp: "Formatted at 2026-06-20 14:47:54 on dist-test-slave-cx89"
server_key: "81cdd2bdab84abaf230c34f8d38b30d5"
server_key_iv: "733c61f15731d91ed4f2df80edf94d42"
server_key_version: "kuduclusterkey@0"
I20260620 14:47:54.049816 11970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:54.065835 11970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:54.068987 11970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:54.069202 11970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:54.069497 11970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:54.069885 11970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:54.069936 11970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:54.069967 11970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:54.070026 11970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:54.077584 11970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34365
I20260620 14:47:54.077661 12098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34365 every 8 connection(s)
I20260620 14:47:54.078115 11970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:47:54.085593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 11970
I20260620 14:47:54.085702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:47:54.085903 392 external_mini_cluster.cc:1428] Setting key abe7f89781ae818509261ed2f9a11aff
Jun 20 14:47:54 dist-test-slave-cx89 krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:47:54.097280 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:54.079284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57785 (local address 127.0.98.62:39689)
0620 14:47:54.079327 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:54.079329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:54.086553 (+ 7224us) server_negotiation.cc:407] Connection header received
0620 14:47:54.087285 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:54.087286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:54.087298 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:54.087358 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:54.088200 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.088984 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.090454 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.090763 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.090804 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:54.093806 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:54.093808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:54.093808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:54.093817 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:54.095513 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:54.096035 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:54.096036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:54.096037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:54.096096 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:54.096371 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:54.096372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:54.096372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:54.096510 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:54.096558 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:54.097084 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 14:47:54.097178 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:47:54.098151 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:54.079016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39689 (local address 127.0.98.3:57785)
0620 14:47:54.086321 (+ 7305us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:54.086388 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:54.087176 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:54.087459 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:54.087464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:54.087714 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:54.088094 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:54.088102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.089084 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.089085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:54.090354 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:54.090356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.090890 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.090891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:54.091031 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:54.091689 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:54.091699 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:54.093701 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:54.095610 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:54.095618 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:54.095621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:54.095943 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:54.096164 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:54.096164 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:54.096165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:54.096298 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:54.096590 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:54.096596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:54.096783 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 14:47:54.097674 (+ 891us) client_negotiation.cc:240] Negotiation successful
0620 14:47:54.097860 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7244,"thread_start_us":80,"threads_started":1}
I20260620 14:47:54.098801 12099 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39689
I20260620 14:47:54.098909 12099 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:54.099099 12099 heartbeater.cc:511] Master 127.0.98.62:39689 requested a full tablet report, sending...
I20260620 14:47:54.099443 11615 ts_manager.cc:194] Registered new tserver with Master: 38b4b4b68b6149cd82a77dc6814f70da (127.0.98.3:34365)
I20260620 14:47:54.099917 11615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57785
I20260620 14:47:54.108758 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:47:54.118448 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:54.110389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50516 (local address 127.0.98.62:39689)
0620 14:47:54.110431 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:54.110433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:54.111102 (+ 669us) server_negotiation.cc:407] Connection header received
0620 14:47:54.111160 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:54.111161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:54.111172 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:54.111228 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:54.111967 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.112733 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.113828 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.114121 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.114187 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:54.115676 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:54.115678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:54.115678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:54.115685 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:54.117260 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:54.117643 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:54.117645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:54.117645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:54.117713 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:54.117926 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:54.117926 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:54.117927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:54.118054 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:54.118149 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:54.118273 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 14:47:54.118355 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:47:54.120416 11615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:47:54.122494 11615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:47:54.140095 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:54.127756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34365 (local address 127.0.0.1:44144)
0620 14:47:54.135183 (+ 7427us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:54.135189 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:54.135234 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:54.135525 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:54.135526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:54.135527 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:54.135581 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:54.135582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.137140 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.137140 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:54.138282 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:54.138284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.139554 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.139555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:54.139808 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:54.139840 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 14:47:54.139937 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 14:47:54.140001 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7417,"thread_start_us":48,"threads_started":1}
I20260620 14:47:54.140434 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:54.127960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44144 (local address 127.0.98.3:34365)
0620 14:47:54.131237 (+ 3277us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:54.131243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:54.135287 (+ 4044us) server_negotiation.cc:407] Connection header received
0620 14:47:54.135332 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:54.135335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:54.135350 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:54.135442 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:54.135647 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.137056 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.138388 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.139128 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.139192 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:54.139278 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:54.140292 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0620 14:47:54.140334 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3234,"thread_start_us":74,"threads_started":1}
I20260620 14:47:54.141431 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:54.126575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.0.98.2:41223)
0620 14:47:54.126739 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:54.126744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:54.131666 (+ 4922us) server_negotiation.cc:407] Connection header received
0620 14:47:54.131823 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:54.131825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:54.131842 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:54.131892 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:54.132166 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.133563 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.139352 (+ 5789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.140846 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.140891 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:54.140963 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:54.141299 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 14:47:54.141345 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260620 14:47:54.141337 12015 tablet_service.cc:1467] Processing CreateTablet for tablet 0a06cb2c89694b439094351fecd9d867 (DEFAULT_TABLE table=test-table [id=27385582ae3242249a600b081dbf365a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:54.141705 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a06cb2c89694b439094351fecd9d867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:54.141921 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:54.126383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41223 (local address 127.0.0.1:58698)
0620 14:47:54.131385 (+ 5002us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:54.131591 (+ 206us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:54.131709 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:54.132007 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:54.132010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:54.132013 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:54.132094 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:54.132096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.133643 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.133643 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:54.134961 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:54.134963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.141137 (+ 6174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.141138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:54.141211 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:54.141215 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:47:54.141709 (+ 494us) client_negotiation.cc:240] Negotiation successful
0620 14:47:54.141813 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4950,"thread_start_us":79,"threads_started":1}
I20260620 14:47:54.142683 11893 tablet_service.cc:1467] Processing CreateTablet for tablet 0a06cb2c89694b439094351fecd9d867 (DEFAULT_TABLE table=test-table [id=27385582ae3242249a600b081dbf365a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:54.142983 11893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a06cb2c89694b439094351fecd9d867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:54.144766 12117 tablet_bootstrap.cc:492] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da: Bootstrap starting.
I20260620 14:47:54.145766 12117 tablet_bootstrap.cc:654] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:54.146046 12117 log.cc:824] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:47:54 dist-test-slave-cx89 krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:47:54.147204 12119 tablet_bootstrap.cc:492] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a: Bootstrap starting.
I20260620 14:47:54.147495 12117 tablet_bootstrap.cc:492] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da: No bootstrap required, opened a new log
I20260620 14:47:54.147565 12117 ts_tablet_manager.cc:1397] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:47:54.148026 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:54.127603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33447 (local address 127.0.0.1:34422)
0620 14:47:54.135074 (+ 7471us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:54.135083 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:54.135120 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:54.140996 (+ 5876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:54.140997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:54.141000 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:54.141050 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:54.141051 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.142213 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.142214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:54.143235 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:54.143237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.147789 (+ 4552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.147790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:54.147864 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:54.147868 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:47:54.147913 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 14:47:54.147955 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7451,"thread_start_us":70,"threads_started":1}
I20260620 14:47:54.148361 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:54.127695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34422 (local address 127.0.98.1:33447)
0620 14:47:54.139504 (+ 11809us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:54.139509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:54.139519 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:47:54.139553 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:54.139555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:54.139571 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:54.139646 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:54.141112 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.142131 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.146891 (+ 4760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.147616 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.147686 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:54.147752 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:54.148247 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 14:47:54.148291 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11767,"thread_start_us":53,"threads_started":1}
I20260620 14:47:54.148859 12119 tablet_bootstrap.cc:654] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:54.149173 12119 log.cc:824] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:54.149410 12117 raft_consensus.cc:348] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bca1be0032c484eb868b7ba055fdfba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33447 } } peers { permanent_uuid: "03eb284351264d88a4aa5016829a9f5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41223 } } peers { permanent_uuid: "38b4b4b68b6149cd82a77dc6814f70da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34365 } }
I20260620 14:47:54.149576 12117 raft_consensus.cc:374] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:54.149611 12117 raft_consensus.cc:729] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b4b4b68b6149cd82a77dc6814f70da, State: Initialized, Role: FOLLOWER
I20260620 14:47:54.149717 12117 consensus_queue.cc:260] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bca1be0032c484eb868b7ba055fdfba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33447 } } peers { permanent_uuid: "03eb284351264d88a4aa5016829a9f5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41223 } } peers { permanent_uuid: "38b4b4b68b6149cd82a77dc6814f70da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34365 } }
I20260620 14:47:54.149971 12117 ts_tablet_manager.cc:1428] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:47:54.150115 12099 heartbeater.cc:503] Master 127.0.98.62:39689 was elected leader, sending a full tablet report...
I20260620 14:47:54.150142 11749 tablet_service.cc:1467] Processing CreateTablet for tablet 0a06cb2c89694b439094351fecd9d867 (DEFAULT_TABLE table=test-table [id=27385582ae3242249a600b081dbf365a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:54.150468 11749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a06cb2c89694b439094351fecd9d867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:54.153689 12122 tablet_bootstrap.cc:492] T 0a06cb2c89694b439094351fecd9d867 P 1bca1be0032c484eb868b7ba055fdfba: Bootstrap starting.
I20260620 14:47:54.156486 12119 tablet_bootstrap.cc:492] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a: No bootstrap required, opened a new log
I20260620 14:47:54.156558 12119 ts_tablet_manager.cc:1397] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260620 14:47:54.158421 12119 raft_consensus.cc:348] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bca1be0032c484eb868b7ba055fdfba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33447 } } peers { permanent_uuid: "03eb284351264d88a4aa5016829a9f5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41223 } } peers { permanent_uuid: "38b4b4b68b6149cd82a77dc6814f70da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34365 } }
I20260620 14:47:54.158558 12119 raft_consensus.cc:374] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:54.158588 12119 raft_consensus.cc:729] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03eb284351264d88a4aa5016829a9f5a, State: Initialized, Role: FOLLOWER
I20260620 14:47:54.158707 12119 consensus_queue.cc:260] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bca1be0032c484eb868b7ba055fdfba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33447 } } peers { permanent_uuid: "03eb284351264d88a4aa5016829a9f5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41223 } } peers { permanent_uuid: "38b4b4b68b6149cd82a77dc6814f70da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34365 } }
I20260620 14:47:54.158986 12119 ts_tablet_manager.cc:1428] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:47:54.158998 11965 heartbeater.cc:503] Master 127.0.98.62:39689 was elected leader, sending a full tablet report...
I20260620 14:47:54.161001 12122 tablet_bootstrap.cc:654] T 0a06cb2c89694b439094351fecd9d867 P 1bca1be0032c484eb868b7ba055fdfba: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:54.161341 12122 log.cc:824] T 0a06cb2c89694b439094351fecd9d867 P 1bca1be0032c484eb868b7ba055fdfba: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:54.163740 12122 tablet_bootstrap.cc:492] T 0a06cb2c89694b439094351fecd9d867 P 1bca1be0032c484eb868b7ba055fdfba: No bootstrap required, opened a new log
I20260620 14:47:54.163817 12122 ts_tablet_manager.cc:1397] T 0a06cb2c89694b439094351fecd9d867 P 1bca1be0032c484eb868b7ba055fdfba: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260620 14:47:54.166337 12122 raft_consensus.cc:348] T 0a06cb2c89694b439094351fecd9d867 P 1bca1be0032c484eb868b7ba055fdfba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bca1be0032c484eb868b7ba055fdfba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33447 } } peers { permanent_uuid: "03eb284351264d88a4aa5016829a9f5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41223 } } peers { permanent_uuid: "38b4b4b68b6149cd82a77dc6814f70da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34365 } }
I20260620 14:47:54.166517 12122 raft_consensus.cc:374] T 0a06cb2c89694b439094351fecd9d867 P 1bca1be0032c484eb868b7ba055fdfba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:54.166566 12122 raft_consensus.cc:729] T 0a06cb2c89694b439094351fecd9d867 P 1bca1be0032c484eb868b7ba055fdfba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bca1be0032c484eb868b7ba055fdfba, State: Initialized, Role: FOLLOWER
I20260620 14:47:54.166690 12122 consensus_queue.cc:260] T 0a06cb2c89694b439094351fecd9d867 P 1bca1be0032c484eb868b7ba055fdfba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bca1be0032c484eb868b7ba055fdfba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33447 } } peers { permanent_uuid: "03eb284351264d88a4aa5016829a9f5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41223 } } peers { permanent_uuid: "38b4b4b68b6149cd82a77dc6814f70da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34365 } }
I20260620 14:47:54.167356 12122 ts_tablet_manager.cc:1428] T 0a06cb2c89694b439094351fecd9d867 P 1bca1be0032c484eb868b7ba055fdfba: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:47:54.167506 11827 heartbeater.cc:503] Master 127.0.98.62:39689 was elected leader, sending a full tablet report...
W20260620 14:47:54.321436 11966 tablet.cc:2369] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:47:54.329047 12100 tablet.cc:2369] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:47:54.349437 11828 tablet.cc:2369] T 0a06cb2c89694b439094351fecd9d867 P 1bca1be0032c484eb868b7ba055fdfba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:47:54.353462 12123 raft_consensus.cc:482] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:47:54.353592 12123 raft_consensus.cc:504] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bca1be0032c484eb868b7ba055fdfba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33447 } } peers { permanent_uuid: "03eb284351264d88a4aa5016829a9f5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41223 } } peers { permanent_uuid: "38b4b4b68b6149cd82a77dc6814f70da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34365 } }
I20260620 14:47:54.354025 12123 leader_election.cc:288] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bca1be0032c484eb868b7ba055fdfba (127.0.98.1:33447), 38b4b4b68b6149cd82a77dc6814f70da (127.0.98.3:34365)
I20260620 14:47:54.359570 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:54.354710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59637 (local address 127.0.98.1:33447)
0620 14:47:54.354772 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:54.354774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:54.355836 (+ 1062us) server_negotiation.cc:407] Connection header received
0620 14:47:54.355936 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:54.355937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:54.355949 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:54.355974 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:54.356160 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.356868 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.358652 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.359194 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.359235 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:54.359245 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:54.359458 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 14:47:54.359499 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:47:54.359987 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:54.354335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33447 (local address 127.0.98.2:59637)
0620 14:47:54.355760 (+ 1425us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:54.355772 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:54.355868 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:54.356034 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:54.356035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:54.356037 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:54.356100 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:54.356102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.357317 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.357318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:54.358555 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:54.358558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.359299 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.359300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:54.359363 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:54.359366 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:47:54.359865 (+ 499us) client_negotiation.cc:240] Negotiation successful
0620 14:47:54.359910 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1400,"thread_start_us":74,"threads_started":1}
I20260620 14:47:54.360411 11781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a06cb2c89694b439094351fecd9d867" candidate_uuid: "03eb284351264d88a4aa5016829a9f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bca1be0032c484eb868b7ba055fdfba" is_pre_election: true
I20260620 14:47:54.360589 11781 raft_consensus.cc:2454] T 0a06cb2c89694b439094351fecd9d867 P 1bca1be0032c484eb868b7ba055fdfba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03eb284351264d88a4aa5016829a9f5a in term 0.
I20260620 14:47:54.360831 11854 leader_election.cc:302] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03eb284351264d88a4aa5016829a9f5a, 1bca1be0032c484eb868b7ba055fdfba; no voters:
I20260620 14:47:54.360952 12123 raft_consensus.cc:2781] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:47:54.360985 12123 raft_consensus.cc:482] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:47:54.361003 12123 raft_consensus.cc:3037] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:54.361824 12123 raft_consensus.cc:504] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bca1be0032c484eb868b7ba055fdfba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33447 } } peers { permanent_uuid: "03eb284351264d88a4aa5016829a9f5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41223 } } peers { permanent_uuid: "38b4b4b68b6149cd82a77dc6814f70da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34365 } }
I20260620 14:47:54.361948 12123 leader_election.cc:288] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [CANDIDATE]: Term 1 election: Requested vote from peers 1bca1be0032c484eb868b7ba055fdfba (127.0.98.1:33447), 38b4b4b68b6149cd82a77dc6814f70da (127.0.98.3:34365)
I20260620 14:47:54.362221 11781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a06cb2c89694b439094351fecd9d867" candidate_uuid: "03eb284351264d88a4aa5016829a9f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bca1be0032c484eb868b7ba055fdfba"
I20260620 14:47:54.362293 11781 raft_consensus.cc:3037] T 0a06cb2c89694b439094351fecd9d867 P 1bca1be0032c484eb868b7ba055fdfba [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:54.363061 11781 raft_consensus.cc:2454] T 0a06cb2c89694b439094351fecd9d867 P 1bca1be0032c484eb868b7ba055fdfba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03eb284351264d88a4aa5016829a9f5a in term 1.
I20260620 14:47:54.363262 11854 leader_election.cc:302] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03eb284351264d88a4aa5016829a9f5a, 1bca1be0032c484eb868b7ba055fdfba; no voters:
I20260620 14:47:54.363363 12123 raft_consensus.cc:2781] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:54.363544 12123 raft_consensus.cc:686] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [term 1 LEADER]: Becoming Leader. State: Replica: 03eb284351264d88a4aa5016829a9f5a, State: Running, Role: LEADER
I20260620 14:47:54.363637 12123 consensus_queue.cc:237] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bca1be0032c484eb868b7ba055fdfba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33447 } } peers { permanent_uuid: "03eb284351264d88a4aa5016829a9f5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41223 } } peers { permanent_uuid: "38b4b4b68b6149cd82a77dc6814f70da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34365 } }
I20260620 14:47:54.364372 11615 catalog_manager.cc:5495] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 03eb284351264d88a4aa5016829a9f5a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "03eb284351264d88a4aa5016829a9f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bca1be0032c484eb868b7ba055fdfba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03eb284351264d88a4aa5016829a9f5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38b4b4b68b6149cd82a77dc6814f70da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34365 } health_report { overall_health: UNKNOWN } } }
I20260620 14:47:54.367928 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:54.354777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59611 (local address 127.0.98.3:34365)
0620 14:47:54.354823 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:54.354825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:54.360360 (+ 5535us) server_negotiation.cc:407] Connection header received
0620 14:47:54.360457 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:54.360458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:54.360469 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:54.360496 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:54.365167 (+ 4671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.365878 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.367066 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.367580 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.367626 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:54.367636 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:54.367825 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 14:47:54.367862 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:47:54.368223 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:54.354515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34365 (local address 127.0.98.2:59611)
0620 14:47:54.360311 (+ 5796us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:54.360321 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:54.360386 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:54.365036 (+ 4650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:54.365037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:54.365039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:54.365074 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:54.365075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.365983 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.365984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:54.366976 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:54.366978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.367685 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.367685 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:54.367753 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:54.367756 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:47:54.368121 (+ 365us) client_negotiation.cc:240] Negotiation successful
0620 14:47:54.368161 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5784,"thread_start_us":52,"threads_started":1}
I20260620 14:47:54.368661 12053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a06cb2c89694b439094351fecd9d867" candidate_uuid: "03eb284351264d88a4aa5016829a9f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b4b4b68b6149cd82a77dc6814f70da" is_pre_election: true
I20260620 14:47:54.368817 12053 raft_consensus.cc:2454] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03eb284351264d88a4aa5016829a9f5a in term 0.
I20260620 14:47:54.368932 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a06cb2c89694b439094351fecd9d867" candidate_uuid: "03eb284351264d88a4aa5016829a9f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b4b4b68b6149cd82a77dc6814f70da"
I20260620 14:47:54.368985 12052 raft_consensus.cc:3037] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:54.369784 12052 raft_consensus.cc:2454] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03eb284351264d88a4aa5016829a9f5a in term 1.
I20260620 14:47:54.420174 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:54.417087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58712 (local address 127.0.98.2:41223)
0620 14:47:54.417126 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:54.417128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:54.417135 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:47:54.417174 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:54.417174 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:54.417197 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:54.417255 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:54.417447 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.418128 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.419128 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.419391 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.419496 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:54.419677 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:54.419809 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:54.419872 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:54.420078 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 14:47:54.420098 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:47:54.428148 12052 raft_consensus.cc:1261] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da [term 1 FOLLOWER]: Refusing update from remote peer 03eb284351264d88a4aa5016829a9f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:54.428148 11781 raft_consensus.cc:1261] T 0a06cb2c89694b439094351fecd9d867 P 1bca1be0032c484eb868b7ba055fdfba [term 1 FOLLOWER]: Refusing update from remote peer 03eb284351264d88a4aa5016829a9f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:54.428517 12128 consensus_queue.cc:1035] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bca1be0032c484eb868b7ba055fdfba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:54.428829 12128 consensus_queue.cc:1035] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b4b4b68b6149cd82a77dc6814f70da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:54.436427 12131 mvcc.cc:205] Tried to move back new op lower bound from 7298936317652353024 to 7298936317393993728. Current Snapshot: MvccSnapshot[applied={T|T < 7298936317652353024}]
I20260620 14:47:54.442509 11615 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:54.442624 11615 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:54.443272 11615 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0baf7ae3146c4bd0a217e36f0c7d8ad2: Sending DeleteTablet for 3 replicas of tablet 0a06cb2c89694b439094351fecd9d867
I20260620 14:47:54.444063 11893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a06cb2c89694b439094351fecd9d867 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58698
I20260620 14:47:54.444499 11749 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a06cb2c89694b439094351fecd9d867 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34422
I20260620 14:47:54.444742 12015 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a06cb2c89694b439094351fecd9d867 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44144
I20260620 14:47:54.444844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 11696
I20260620 14:47:54.450603 12141 tablet_replica.cc:330] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da: stopping tablet replica
I20260620 14:47:54.451537 12141 raft_consensus.cc:2229] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:54.451732 12141 pending_rounds.cc:62] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da: Trying to abort 2 pending ops.
I20260620 14:47:54.451807 12141 pending_rounds.cc:68] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7298936317652353024 op_type: WRITE_OP write_request { tablet_id: "0a06cb2c89694b439094351fecd9d867" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\306\322\332\321\006\"8\n\ntest-admin\022*\n 27385582ae3242249a600b081dbf365a\020\001\030\001 \001(\001" signature: "<redacted>""#\006PHv\353\026/a\035\375#_\374\330~\355K \264w4\312\035\005V\374\361\274\236\217}\331Ca\003\366P\376\2610\205\274\371\337\356\236]\364\277E\255\206\033\226u\333\"\371\rz\244\353\375" signing_key_seq_num: 0 } } request_id { client_id: "21a4c36ae67e48f0b40e0d184b56bd20" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260620 14:47:54.452338 12141 pending_rounds.cc:68] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7298936317393993728 op_type: NO_OP noop_request { }
W20260620 14:47:54.452561 11854 consensus_peers.cc:487] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a -> Peer 38b4b4b68b6149cd82a77dc6814f70da (127.0.98.3:34365): Couldn't send request to peer 38b4b4b68b6149cd82a77dc6814f70da. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:47:54.453001 12141 raft_consensus.cc:2866] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260620 14:47:54.453125 12141 raft_consensus.cc:2258] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:54.454365 12141 ts_tablet_manager.cc:1905] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:47:54.458235 11854 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33447 (error 108)
I20260620 14:47:54.458475 12141 ts_tablet_manager.cc:1918] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:47:54.458623 12141 log.cc:1196] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/0a06cb2c89694b439094351fecd9d867
I20260620 14:47:54.459209 12141 ts_tablet_manager.cc:1939] T 0a06cb2c89694b439094351fecd9d867 P 38b4b4b68b6149cd82a77dc6814f70da: Deleting consensus metadata
I20260620 14:47:54.459824 11602 catalog_manager.cc:4841] TS 38b4b4b68b6149cd82a77dc6814f70da (127.0.98.3:34365): tablet 0a06cb2c89694b439094351fecd9d867 (table test-table [id=27385582ae3242249a600b081dbf365a]) successfully deleted
W20260620 14:47:54.460466 11602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33447 (error 108)
I20260620 14:47:54.461011 12139 tablet_replica.cc:330] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a: stopping tablet replica
I20260620 14:47:54.461150 12139 raft_consensus.cc:2229] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:47:54.461305 12139 raft_consensus.cc:2258] T 0a06cb2c89694b439094351fecd9d867 P 03eb284351264d88a4aa5016829a9f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:54.461679 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:54.461571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33447 (local address 127.0.98.2:32999)
0620 14:47:54.461616 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:54.461647 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 14:47:54.461860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 11835
Jun 20 14:47:54 dist-test-slave-cx89 krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:47:54.464918 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:54.464704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33447 (local address 127.0.0.1:34426)
0620 14:47:54.464758 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:54.464878 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 14:47:54.465080 11602 catalog_manager.cc:4584] TS 1bca1be0032c484eb868b7ba055fdfba (127.0.98.1:33447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a06cb2c89694b439094351fecd9d867: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33447: connect: Connection refused (error 111)
I20260620 14:47:54.468497 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:54.468385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41223 (local address 127.0.0.1:58726)
0620 14:47:54.468426 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:54.468461 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 14:47:54.468525 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 11970
I20260620 14:47:54.473520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 11583
I20260620 14:47:54.478881 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:47:54.813058 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:47:54.813144 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:47:55.147262 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:47:55.147 UTC [11169] LOG: received smart shutdown request
2026-06-20 14:47:55.149 UTC [11169] LOG: background worker "logical replication launcher" (PID 11178) exited with exit code 1
2026-06-20 14:47:55.150 UTC [11173] LOG: shutting down
2026-06-20 14:47:55.313 UTC [11169] LOG: database system is shut down
2026-06-20T14:47:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43276 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:47:55 dist-test-slave-cx89 krb5kdc[12152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:47:55 dist-test-slave-cx89 krb5kdc[12152](info): set up 2 sockets
Jun 20 14:47:55 dist-test-slave-cx89 krb5kdc[12152](info): commencing operation
krb5kdc: starting...
W20260620 14:47:56.351027 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:47:56 dist-test-slave-cx89 krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:47:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 14:47:56.500074 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskV7d6BE/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/postgres -l logfile start
2026-06-20 14:47:58.790 UTC [12182] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:47:58.790 UTC [12182] LOG: listening on IPv4 address "127.0.98.20", port 56769
2026-06-20 14:47:58.793 UTC [12182] LOG: listening on Unix socket "/tmp/.s.PGSQL.56769"
2026-06-20 14:47:58.820 UTC [12185] LOG: database system was shut down at 2026-06-20 14:47:57 UTC
2026-06-20 14:47:58.823 UTC [12182] LOG: database system is ready to accept connections
I20260620 14:47:59.694928 392 mini_postgres.cc:96] Postgres bound to 56769
2026-06-20 14:47:59.700 UTC [12193] FATAL: database "slave" does not exist
127.0.98.20:56769 - accepting connections
I20260620 14:47:59.700712 392 mini_ranger.cc:161] Starting Ranger...
I20260620 14:47:59.714524 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:48:00.140759 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:48:00.140825 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ranger-admin
2026-06-20 14:48:00,398 [I] DB FLAVOR :POSTGRES
2026-06-20 14:48:00,398 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:48:00,398 [I] Checking connection..
2026-06-20 14:48:00,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:48:00,655 [I] Checking connection passed.
2026-06-20 14:48:00,655 [I] --------- Verifying version history table ---------
2026-06-20 14:48:00,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:48:00,901 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:48:00,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:48:01,131 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:48:01,132 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:48:01,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:48:01.379 UTC [12257] WARNING: there is no transaction in progress
2026-06-20 14:48:01,385 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:48:01,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:48:01,618 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:48:01,618 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:48:01,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:48:01,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:48:02,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cx89.c.gcp-upstream.internal','N') ;"
2026-06-20 14:48:02,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:48:02,571 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:48:02,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:48:02,796 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:48:02,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:48:03,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:48:03,278 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:48:03,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:48:04.184 UTC [12365] WARNING: there is no transaction in progress
2026-06-20 14:48:04.201 UTC [12365] WARNING: there is no transaction in progress
2026-06-20 14:48:04.218 UTC [12365] WARNING: there is no transaction in progress
2026-06-20 14:48:04.235 UTC [12365] WARNING: there is no transaction in progress
2026-06-20 14:48:04.250 UTC [12365] WARNING: there is no transaction in progress
2026-06-20 14:48:04.268 UTC [12365] WARNING: there is no transaction in progress
2026-06-20 14:48:04.321 UTC [12365] WARNING: there is no transaction in progress
2026-06-20 14:48:04.328 UTC [12365] WARNING: there is no transaction in progress
2026-06-20 14:48:04.342 UTC [12365] WARNING: there is no transaction in progress
2026-06-20 14:48:04.352 UTC [12365] WARNING: there is no transaction in progress
2026-06-20 14:48:04.362 UTC [12365] WARNING: there is no transaction in progress
2026-06-20 14:48:04.374 UTC [12365] WARNING: there is no transaction in progress
2026-06-20 14:48:04.384 UTC [12365] WARNING: there is no transaction in progress
2026-06-20 14:48:04.390 UTC [12365] WARNING: there is no transaction in progress
2026-06-20 14:48:04.396 UTC [12365] WARNING: there is no transaction in progress
2026-06-20 14:48:04.402 UTC [12365] WARNING: there is no transaction in progress
2026-06-20 14:48:05,006 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:48:05,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:48:05,222 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:48:05,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:48:05,479 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:48:05,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:48:05,704 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:48:05,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:48:05,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:48:06,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cx89.c.gcp-upstream.internal';"
2026-06-20 14:48:06,408 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:48:06,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:48:06,657 [I] DB_PATCHES have already been applied
I20260620 14:48:06.663055 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskV7d6BE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:48:06.663129 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 14:48:06.665870 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:46233
Jun 20, 2026 2:48:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:48:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:48:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:48:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:48:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:48:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:48:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:48:07.665174 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.964s user 0.000s sys 0.007s
Jun 20, 2026 2:48:27 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:48:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 2:48:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:48:27 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:48:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:48:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:48:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:48:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:48:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:48:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:48:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:48:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:48:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:48:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:48:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:48:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:48:27 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:48:28 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:48:28.655913 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:48:28 dist-test-slave-cx89 krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966908, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:48:28.727895 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:48:28.736466 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:48:29.480085 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:48:29.480150 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ranger-kms
I20260620 14:48:29.480167 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:56769
2026-06-20 14:48:29,622 [I] DB FLAVOR :POSTGRES
2026-06-20 14:48:29,622 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:48:29,622 [I] Checking connection
2026-06-20 14:48:29,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:48:29,876 [I] connection success
2026-06-20 14:48:29,876 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:48:29,876 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:48:29,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:48:30,099 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:48:30,099 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:48:30,099 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:48:30,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56769/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:48:30,341 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:48:30.781824 392 mini_ranger_kms.cc:310] Created kms service
I20260620 14:48:30.957909 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:48:31.011339 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:48:31 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966908, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:48:31.426906 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:48:31.427042 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskV7d6BE/build/release/bin/postgresql.jar:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskV7d6BE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:48:31.427064 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 14:48:31.430200 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12545
I20260620 14:48:31.430272 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:52463
Jun 20, 2026 2:48:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:48:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:48:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:48:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:48:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:48:32 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 14:48:33.420686 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.693s user 0.003s sys 0.123s
I20260620 14:48:33.420766 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:48:33.420810 392 mini_ranger_kms.cc:401] 127.0.98.20:52463/kms/v1/keys
Jun 20 14:48:33 dist-test-slave-cx89 krb5kdc[12152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781966913, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 14:48:34 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966913, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:48:34 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966913, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:48:36 dist-test-slave-cx89 krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:48:36.833837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41641
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41641
--ranger_config_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52463/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:36.913614 12596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:36.913817 12596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:36.913837 12596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:36.914930 12596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:36.914970 12596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:36.914983 12596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:36.914994 12596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:36.915004 12596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:36.916584 12596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:52463/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41641
--ranger_config_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:36.916800 12596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:36.917003 12596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:36.922049 12604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:36.922055 12602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:36.922056 12601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:36.922219 12596 server_base.cc:1034] running on GCE node
I20260620 14:48:36.922591 12596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:36.922883 12596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:36.924023 12596 hybrid_clock.cc:648] HybridClock initialized: now 1781966916924004 us; error 36 us; skew 500 ppm
Jun 20 14:48:36 dist-test-slave-cx89 krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:36.926509 12596 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:48:36.926946 12596 webserver.cc:460] Webserver started at http://127.0.98.62:46699/ using document root <none> and password file <none>
I20260620 14:48:36.927137 12596 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:36.927181 12596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:36.927312 12596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:48:36 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:48:36.971056 12596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "20e408e734d84575a2f0364dc8f0f1d7"
format_stamp: "Formatted at 2026-06-20 14:48:36 on dist-test-slave-cx89"
server_key: "7a05543e73a015a120aee17566846b51"
server_key_iv: "b8d0cf78b096321dd10fda5af5e1f759"
server_key_version: "kuduclusterkey@0"
I20260620 14:48:36.971534 12596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "20e408e734d84575a2f0364dc8f0f1d7"
format_stamp: "Formatted at 2026-06-20 14:48:36 on dist-test-slave-cx89"
server_key: "7a05543e73a015a120aee17566846b51"
server_key_iv: "b8d0cf78b096321dd10fda5af5e1f759"
server_key_version: "kuduclusterkey@0"
I20260620 14:48:36.973213 12596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:36.984491 12611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:36.984694 12596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:36.984786 12596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "20e408e734d84575a2f0364dc8f0f1d7"
format_stamp: "Formatted at 2026-06-20 14:48:36 on dist-test-slave-cx89"
server_key: "7a05543e73a015a120aee17566846b51"
server_key_iv: "b8d0cf78b096321dd10fda5af5e1f759"
server_key_version: "kuduclusterkey@0"
I20260620 14:48:36.984875 12596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:36.993618 12596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:36.995559 12596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:36.995684 12596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:36.999457 12596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41641
I20260620 14:48:36.999547 12663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41641 every 8 connection(s)
I20260620 14:48:36.999859 12596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:37.000566 12664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:37.003475 12664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7: Bootstrap starting.
I20260620 14:48:37.004130 12664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:37.004434 12664 log.cc:824] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:37.005131 12664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7: No bootstrap required, opened a new log
I20260620 14:48:37.007805 12664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e408e734d84575a2f0364dc8f0f1d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41641 } }
I20260620 14:48:37.007989 12664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:37.008019 12664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e408e734d84575a2f0364dc8f0f1d7, State: Initialized, Role: FOLLOWER
I20260620 14:48:37.008167 12664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e408e734d84575a2f0364dc8f0f1d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41641 } }
I20260620 14:48:37.008232 12664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:37.008258 12664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:37.008293 12664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:37.008934 12664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e408e734d84575a2f0364dc8f0f1d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41641 } }
I20260620 14:48:37.009006 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 12596
I20260620 14:48:37.009039 12664 leader_election.cc:302] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20e408e734d84575a2f0364dc8f0f1d7; no voters:
I20260620 14:48:37.009091 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:48:37.009197 12664 leader_election.cc:288] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:37.009276 392 external_mini_cluster.cc:1428] Setting key 502f7e14598a3f8b0a84cb5f4cae417b
I20260620 14:48:37.009316 12667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:37.009451 12667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [term 1 LEADER]: Becoming Leader. State: Replica: 20e408e734d84575a2f0364dc8f0f1d7, State: Running, Role: LEADER
I20260620 14:48:37.009549 12667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e408e734d84575a2f0364dc8f0f1d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41641 } }
I20260620 14:48:37.009801 12664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:37.009861 12667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20e408e734d84575a2f0364dc8f0f1d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e408e734d84575a2f0364dc8f0f1d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41641 } } }
I20260620 14:48:37.009955 12667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:37.010085 12667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20e408e734d84575a2f0364dc8f0f1d7. Latest consensus state: current_term: 1 leader_uuid: "20e408e734d84575a2f0364dc8f0f1d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e408e734d84575a2f0364dc8f0f1d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41641 } } }
I20260620 14:48:37.010156 12667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:37.010296 12673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:37.010977 12673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:37.011245 12664 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:48:37.016244 12673 catalog_manager.cc:1347] Generated new cluster ID: 82a9af8e88574b3d9e2472b749030d54
I20260620 14:48:37.016433 12673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:48:37 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:48:37.026858 12671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:37.009724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.0.98.62:41641)
0620 14:48:37.009999 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:37.010029 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:37.013262 (+ 3233us) server_negotiation.cc:407] Connection header received
0620 14:48:37.013735 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:37.013744 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:37.013860 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:37.014022 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:37.015371 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.017154 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.018383 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.019162 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.019306 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:37.022420 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:37.022425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:37.022425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:37.022437 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:37.024641 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:37.025025 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:37.025028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:37.025029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:37.025125 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:37.025333 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:37.025334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:37.025334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:37.025696 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:37.025854 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:37.026379 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 14:48:37.026546 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":74,"threads_started":1}
I20260620 14:48:37.032470 12673 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:37.033219 12673 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:37.040977 12673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7: Generated new TSK 0
I20260620 14:48:37.041227 12673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:48:37.756067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52463/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41641
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:37.837989 12709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:37.838184 12709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:37.838213 12709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:37.839336 12709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:37.839390 12709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:37.839444 12709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:48:37.840957 12709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:52463/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:37.841183 12709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:37.841414 12709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:37.846354 12714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:37.846366 12717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:37.846532 12709 server_base.cc:1034] running on GCE node
W20260620 14:48:37.846365 12715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:37.846921 12709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:37.847158 12709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:37.848294 12709 hybrid_clock.cc:648] HybridClock initialized: now 1781966917848280 us; error 32 us; skew 500 ppm
Jun 20 14:48:37 dist-test-slave-cx89 krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:37.850962 12709 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:48:37.851547 12709 webserver.cc:460] Webserver started at http://127.0.98.1:43651/ using document root <none> and password file <none>
I20260620 14:48:37.851815 12709 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:37.851878 12709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:37.852008 12709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:48:37 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:48:37.867122 12709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "12872ae9e3404dc695483e8c0750d2f0"
format_stamp: "Formatted at 2026-06-20 14:48:37 on dist-test-slave-cx89"
server_key: "9d0cebbe731178a1c32fbfe75d779936"
server_key_iv: "09f354ccc676f5f31ce85b265314dcc7"
server_key_version: "kuduclusterkey@0"
I20260620 14:48:37.867594 12709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "12872ae9e3404dc695483e8c0750d2f0"
format_stamp: "Formatted at 2026-06-20 14:48:37 on dist-test-slave-cx89"
server_key: "9d0cebbe731178a1c32fbfe75d779936"
server_key_iv: "09f354ccc676f5f31ce85b265314dcc7"
server_key_version: "kuduclusterkey@0"
I20260620 14:48:37.871807 12709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20260620 14:48:37.884769 12724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:37.884965 12709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:37.885059 12709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "12872ae9e3404dc695483e8c0750d2f0"
format_stamp: "Formatted at 2026-06-20 14:48:37 on dist-test-slave-cx89"
server_key: "9d0cebbe731178a1c32fbfe75d779936"
server_key_iv: "09f354ccc676f5f31ce85b265314dcc7"
server_key_version: "kuduclusterkey@0"
I20260620 14:48:37.885150 12709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:37.900640 12709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:37.902801 12709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:37.902940 12709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:37.903136 12709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:37.903475 12709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:37.903518 12709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:37.903548 12709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:37.903570 12709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:37.909830 12709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36403
I20260620 14:48:37.910064 12839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36403 every 8 connection(s)
I20260620 14:48:37.910244 12709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:37.910707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 12709
I20260620 14:48:37.910804 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:48:37.911013 392 external_mini_cluster.cc:1428] Setting key b726c194593b528be90595cd775db31c
Jun 20 14:48:37 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:48:37.922508 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:37.912323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35683 (local address 127.0.98.62:41641)
0620 14:48:37.912466 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:37.912468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:37.912478 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:48:37.913078 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:37.913080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:37.913094 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:37.913173 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:37.913936 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.914694 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.916250 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.916552 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.916631 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:37.919088 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:37.919090 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:37.919090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:37.919097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:37.920744 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:37.921123 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:37.921124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:37.921125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:37.921185 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:37.921402 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:37.921403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:37.921403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:37.921643 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:37.921710 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:37.922325 (+ 615us) server_negotiation.cc:299] Negotiation successful
0620 14:48:37.922404 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:48:37.923058 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:37.912012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41641 (local address 127.0.98.1:35683)
0620 14:48:37.912190 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:37.912249 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:37.912904 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:37.913313 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:37.913317 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:37.913551 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:37.913833 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:37.913839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.914796 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.914798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:37.916078 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:37.916081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.916667 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.916668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:37.916813 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:37.917403 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:37.917413 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:37.918985 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:37.920850 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:37.920856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:37.920858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:37.921057 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:37.921243 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:37.921243 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:37.921244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:37.921349 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:37.921739 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:37.921746 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:37.921916 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 14:48:37.922024 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:48:37.922172 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260620 14:48:37.923616 12840 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41641
I20260620 14:48:37.923704 12840 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:37.923874 12840 heartbeater.cc:511] Master 127.0.98.62:41641 requested a full tablet report, sending...
I20260620 14:48:37.924429 12628 ts_manager.cc:194] Registered new tserver with Master: 12872ae9e3404dc695483e8c0750d2f0 (127.0.98.1:36403)
I20260620 14:48:37.925300 12628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:48:37.936563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52463/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41641
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:38.035172 12846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:38.035467 12846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:38.035504 12846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:38.037238 12846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:38.037299 12846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:38.037336 12846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:48:38.039496 12846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:52463/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:38.039740 12846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:38.040026 12846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:38.046160 12856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:38.046161 12853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:38.046672 12854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:38.046903 12846 server_base.cc:1034] running on GCE node
I20260620 14:48:38.047101 12846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:38.047348 12846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:38.048503 12846 hybrid_clock.cc:648] HybridClock initialized: now 1781966918048480 us; error 40 us; skew 500 ppm
Jun 20 14:48:38 dist-test-slave-cx89 krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:38.051612 12846 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:48:38.052135 12846 webserver.cc:460] Webserver started at http://127.0.98.2:36487/ using document root <none> and password file <none>
I20260620 14:48:38.052385 12846 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:38.052510 12846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:38.052697 12846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:48:38 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:48:38.068943 12846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "a07d805b812547409cba89b947172651"
format_stamp: "Formatted at 2026-06-20 14:48:38 on dist-test-slave-cx89"
server_key: "fee9f2f102ae635234ceabebf4c1fd95"
server_key_iv: "c4b23c003dfe238ffbcfa0acdb3ce90f"
server_key_version: "kuduclusterkey@0"
I20260620 14:48:38.069416 12846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a07d805b812547409cba89b947172651"
format_stamp: "Formatted at 2026-06-20 14:48:38 on dist-test-slave-cx89"
server_key: "fee9f2f102ae635234ceabebf4c1fd95"
server_key_iv: "c4b23c003dfe238ffbcfa0acdb3ce90f"
server_key_version: "kuduclusterkey@0"
I20260620 14:48:38.070825 12846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
Jun 20 14:48:38 dist-test-slave-cx89 krb5kdc[12152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781966918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:38.092334 12863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:38.092595 12846 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260620 14:48:38.092701 12846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "a07d805b812547409cba89b947172651"
format_stamp: "Formatted at 2026-06-20 14:48:38 on dist-test-slave-cx89"
server_key: "fee9f2f102ae635234ceabebf4c1fd95"
server_key_iv: "c4b23c003dfe238ffbcfa0acdb3ce90f"
server_key_version: "kuduclusterkey@0"
I20260620 14:48:38.092798 12846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:38.112984 12846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:38.115459 12846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:38.115619 12846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:38.115859 12846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:38.116218 12846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:38.116295 12846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:38.116339 12846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:38.116369 12846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:38.122310 12846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35623
I20260620 14:48:38.122802 12846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:38.125273 12976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35623 every 8 connection(s)
I20260620 14:48:38.131843 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 12846
I20260620 14:48:38.131943 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:38.132124 392 external_mini_cluster.cc:1428] Setting key d4c3d8db288449781ee481c1deebd7bf
Jun 20 14:48:38 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:48:38.136443 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:38.125936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60543 (local address 127.0.98.62:41641)
0620 14:48:38.125988 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:38.125990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:38.126270 (+ 280us) server_negotiation.cc:407] Connection header received
0620 14:48:38.126941 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:38.126942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:38.126957 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:38.127024 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:38.127709 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.128490 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.129882 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.130260 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.130340 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:38.133243 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:38.133245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:38.133245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:38.133253 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:38.134874 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:38.135347 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:38.135349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:38.135349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:38.135403 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:38.135711 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:38.135712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:38.135713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:38.135859 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:38.135931 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:38.136261 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 14:48:38.136344 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:48:38.136627 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:38.125943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41641 (local address 127.0.98.2:60543)
0620 14:48:38.126115 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:38.126167 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:38.126818 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:38.127091 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:38.127095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:38.127336 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:38.127623 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:38.127629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.128566 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.128567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:38.129792 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:38.129794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.130352 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.130353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:38.130479 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:38.131039 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:38.131048 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:38.133137 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:38.134969 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:38.134976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:38.134979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:38.135261 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:38.135508 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:38.135509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:38.135509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:38.135641 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:38.135933 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:38.135938 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:38.136123 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 14:48:38.136243 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 14:48:38.136399 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260620 14:48:38.137179 12977 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41641
I20260620 14:48:38.137287 12977 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:38.137470 12977 heartbeater.cc:511] Master 127.0.98.62:41641 requested a full tablet report, sending...
I20260620 14:48:38.137859 12628 ts_manager.cc:194] Registered new tserver with Master: a07d805b812547409cba89b947172651 (127.0.98.2:35623)
I20260620 14:48:38.138324 12628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60543
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:48:38.161602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52463/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41641
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:38.274501 12983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:38.274863 12983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:38.274953 12983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:38.276772 12983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:38.276881 12983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:38.276963 12983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:48:38.279204 12983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:52463/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:38.279711 12983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:38.280016 12983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:48:38.286578 12983 server_base.cc:1034] running on GCE node
W20260620 14:48:38.286780 12991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:38.287134 12988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:38.287322 12989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:38.287685 12983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:38.287973 12983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:38.289129 12983 hybrid_clock.cc:648] HybridClock initialized: now 1781966918289101 us; error 43 us; skew 500 ppm
Jun 20 14:48:38 dist-test-slave-cx89 krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:38.292893 12983 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:48:38.293484 12983 webserver.cc:460] Webserver started at http://127.0.98.3:38745/ using document root <none> and password file <none>
I20260620 14:48:38.293766 12983 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:38.293869 12983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:38.294034 12983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:48:38 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:48:38.313606 12983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "47dbe74c36564c6986911b1fae7bf4e6"
format_stamp: "Formatted at 2026-06-20 14:48:38 on dist-test-slave-cx89"
server_key: "64c349ceb622185633b9e6d374b65b7d"
server_key_iv: "8442636652822556326e11e33dcdbc08"
server_key_version: "kuduclusterkey@0"
I20260620 14:48:38.314029 12983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "47dbe74c36564c6986911b1fae7bf4e6"
format_stamp: "Formatted at 2026-06-20 14:48:38 on dist-test-slave-cx89"
server_key: "64c349ceb622185633b9e6d374b65b7d"
server_key_iv: "8442636652822556326e11e33dcdbc08"
server_key_version: "kuduclusterkey@0"
I20260620 14:48:38.315763 12983 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 14:48:38.330905 12998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:38.331135 12983 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:38.331243 12983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "47dbe74c36564c6986911b1fae7bf4e6"
format_stamp: "Formatted at 2026-06-20 14:48:38 on dist-test-slave-cx89"
server_key: "64c349ceb622185633b9e6d374b65b7d"
server_key_iv: "8442636652822556326e11e33dcdbc08"
server_key_version: "kuduclusterkey@0"
I20260620 14:48:38.331346 12983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:38.350005 12983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:38.352447 12983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:38.352653 12983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:38.352973 12983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:38.353394 12983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:38.353477 12983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:38.353549 12983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:38.353606 12983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:38.361388 12983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45167
I20260620 14:48:38.361756 12983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:38.362231 13111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45167 every 8 connection(s)
I20260620 14:48:38.367761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 12983
Jun 20 14:48:38 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:48:38.367858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:38.368036 392 external_mini_cluster.cc:1428] Setting key 4ee963e49c08327c1993ccf95e9c7157
I20260620 14:48:38.370906 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:38.362507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50433 (local address 127.0.98.62:41641)
0620 14:48:38.362548 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:38.362550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:38.362783 (+ 233us) server_negotiation.cc:407] Connection header received
0620 14:48:38.363281 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:38.363282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:38.363294 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:38.363345 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:38.364011 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.364773 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.365861 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.366058 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.366113 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:38.368224 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:38.368226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:38.368226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:38.368233 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:38.369581 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:38.369936 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:38.369938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:38.369938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:38.369991 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:38.370198 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:38.370199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:38.370199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:38.370314 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:38.370366 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:38.370722 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 14:48:38.370811 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:48:38.371970 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:38.362512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41641 (local address 127.0.98.3:50433)
0620 14:48:38.362666 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:38.362708 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:38.363185 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:38.363445 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:38.363450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:38.363636 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:38.363896 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:38.363902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.364890 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.364891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:38.365772 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:38.365774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.366143 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.366144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:38.366254 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:38.366742 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:38.366752 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:38.368102 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:38.369653 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:38.369660 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:38.369662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:38.369873 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:38.370044 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:38.370045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:38.370045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:38.370145 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:38.370366 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:38.370372 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:38.370545 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 14:48:38.371561 (+ 1016us) client_negotiation.cc:240] Negotiation successful
0620 14:48:38.371740 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":98,"threads_started":1}
I20260620 14:48:38.372628 13112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41641
I20260620 14:48:38.372731 13112 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:38.372905 13112 heartbeater.cc:511] Master 127.0.98.62:41641 requested a full tablet report, sending...
I20260620 14:48:38.373251 12628 ts_manager.cc:194] Registered new tserver with Master: 47dbe74c36564c6986911b1fae7bf4e6 (127.0.98.3:45167)
I20260620 14:48:38.373600 12628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50433
I20260620 14:48:38.379601 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:48:38.380139 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 12709
I20260620 14:48:38.385908 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 12846
I20260620 14:48:38.390841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 12983
I20260620 14:48:38.396940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 12596
I20260620 14:48:38.403050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41641
--webserver_interface=127.0.98.62
--webserver_port=46699
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41641
--ranger_config_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52463/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52463/kms with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:38.481491 13115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:38.481709 13115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:38.481731 13115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:38.482808 13115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:38.482843 13115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:38.482857 13115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:38.482867 13115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:38.482878 13115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:38.484705 13115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52463/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41641
--ranger_config_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:38.484952 13115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:38.485155 13115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:38.489773 13120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:38.489764 13121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:38.489764 13123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:38.490020 13115 server_base.cc:1034] running on GCE node
I20260620 14:48:38.490223 13115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:38.490530 13115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:38.491658 13115 hybrid_clock.cc:648] HybridClock initialized: now 1781966918491646 us; error 30 us; skew 500 ppm
Jun 20 14:48:38 dist-test-slave-cx89 krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:38.494042 13115 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:48:38.494452 13115 webserver.cc:460] Webserver started at http://127.0.98.62:46699/ using document root <none> and password file <none>
I20260620 14:48:38.494637 13115 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:38.494689 13115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:48:38 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:48:38.569038 13115 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:48:38.569700 13131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:38.569860 13115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:38.569947 13115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "20e408e734d84575a2f0364dc8f0f1d7"
format_stamp: "Formatted at 2026-06-20 14:48:36 on dist-test-slave-cx89"
server_key: "7a05543e73a015a120aee17566846b51"
server_key_iv: "b8d0cf78b096321dd10fda5af5e1f759"
server_key_version: "kuduclusterkey@0"
I20260620 14:48:38.570302 13115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:38.587549 13115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:38.589193 13115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:38.589323 13115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:38.593281 13115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41641
I20260620 14:48:38.593324 13183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41641 every 8 connection(s)
I20260620 14:48:38.593664 13115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:38.594842 13184 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:48:38.595505 13184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7: Bootstrap starting.
I20260620 14:48:38.598450 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 13115
I20260620 14:48:38.598698 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52463/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36403
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41641
--webserver_port=43651
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52463/kms with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:48:38.603102 13184 log.cc:824] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:38.604792 13184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:48:38.605029 13184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7: Bootstrap complete.
I20260620 14:48:38.607136 13184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e408e734d84575a2f0364dc8f0f1d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41641 } }
I20260620 14:48:38.607363 13184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e408e734d84575a2f0364dc8f0f1d7, State: Initialized, Role: FOLLOWER
I20260620 14:48:38.607535 13184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e408e734d84575a2f0364dc8f0f1d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41641 } }
I20260620 14:48:38.607615 13184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:38.607666 13184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:38.607712 13184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:48:38.608692 13184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e408e734d84575a2f0364dc8f0f1d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41641 } }
I20260620 14:48:38.608798 13184 leader_election.cc:302] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20e408e734d84575a2f0364dc8f0f1d7; no voters:
I20260620 14:48:38.608952 13184 leader_election.cc:288] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:48:38.609014 13188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:48:38.609184 13184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:38.609211 13188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [term 2 LEADER]: Becoming Leader. State: Replica: 20e408e734d84575a2f0364dc8f0f1d7, State: Running, Role: LEADER
I20260620 14:48:38.609390 13188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e408e734d84575a2f0364dc8f0f1d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41641 } }
I20260620 14:48:38.609670 13189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "20e408e734d84575a2f0364dc8f0f1d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e408e734d84575a2f0364dc8f0f1d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41641 } } }
I20260620 14:48:38.609730 13189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:38.610065 13190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20e408e734d84575a2f0364dc8f0f1d7. Latest consensus state: current_term: 2 leader_uuid: "20e408e734d84575a2f0364dc8f0f1d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e408e734d84575a2f0364dc8f0f1d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41641 } } }
I20260620 14:48:38.610127 13190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:38.610636 13184 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:48:38.610946 13195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:38.613277 13195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:38.613608 13195 catalog_manager.cc:1259] Loaded cluster ID: 82a9af8e88574b3d9e2472b749030d54
I20260620 14:48:38.613710 13195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:38.615590 13195 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:38.615809 13195 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7: Loaded TSK: 0
I20260620 14:48:38.616047 13195 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:48:38.682318 13186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:38.682669 13186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:38.682767 13186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:38.684422 13186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:38.684535 13186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:38.684615 13186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:48:38.686741 13186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52463/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43651
--tserver_master_addrs=127.0.98.62:41641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:38.687198 13186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:38.687556 13186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:38.692209 13222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:38.692214 13221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:38.692540 13224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:38.694873 13186 server_base.cc:1034] running on GCE node
I20260620 14:48:38.695058 13186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:38.695294 13186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:38.696449 13186 hybrid_clock.cc:648] HybridClock initialized: now 1781966918696437 us; error 27 us; skew 500 ppm
Jun 20 14:48:38 dist-test-slave-cx89 krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:38.699146 13186 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:48:38.699599 13186 webserver.cc:460] Webserver started at http://127.0.98.1:43651/ using document root <none> and password file <none>
I20260620 14:48:38.699813 13186 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:38.699875 13186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:48:38 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:48:38.723383 13186 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260620 14:48:38.724236 13232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:38.724416 13186 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:38.724495 13186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "12872ae9e3404dc695483e8c0750d2f0"
format_stamp: "Formatted at 2026-06-20 14:48:37 on dist-test-slave-cx89"
server_key: "9d0cebbe731178a1c32fbfe75d779936"
server_key_iv: "09f354ccc676f5f31ce85b265314dcc7"
server_key_version: "kuduclusterkey@0"
I20260620 14:48:38.724920 13186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:38.757470 13186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:38.759738 13186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:38.759970 13186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:38.764664 13186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:38.765594 13186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:38.765702 13186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:38.765859 13186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:38.765964 13186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:38.774273 13186 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36403
I20260620 14:48:38.774551 13345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36403 every 8 connection(s)
I20260620 14:48:38.774966 13186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:38.783666 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 13186
I20260620 14:48:38.783954 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52463/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35623
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41641
--webserver_port=36487
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52463/kms with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:48:38 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:48:38.799310 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:38.776985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44423 (local address 127.0.98.62:41641)
0620 14:48:38.779542 (+ 2557us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:38.779552 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:38.779593 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:48:38.779739 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:38.779746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:38.779807 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:38.779953 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:38.780735 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.782168 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.783628 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.784237 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.784339 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:38.788718 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:38.788723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:38.788723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:38.788737 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:38.797544 (+ 8807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:38.798065 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:38.798068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:38.798068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:38.798155 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:38.798391 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:38.798392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:38.798393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:38.798622 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:38.798695 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:38.798990 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 14:48:38.799096 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2496,"thread_start_us":71,"threads_started":1}
I20260620 14:48:38.799324 13348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:38.776638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41641 (local address 127.0.98.1:44423)
0620 14:48:38.776849 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:38.776900 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:38.777517 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:38.780095 (+ 2578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:38.780099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:38.780360 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:38.780611 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:38.780617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.782287 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.782288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:38.783541 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:38.783544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.784439 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.784440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:38.784560 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:38.785125 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:38.785135 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:38.788572 (+ 3437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:38.797642 (+ 9070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:38.797652 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:38.797655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:38.797983 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:38.798216 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:38.798217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:38.798217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:38.798334 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:38.798686 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:38.798692 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:38.798841 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 14:48:38.798931 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:48:38.799096 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":85,"threads_started":1}
I20260620 14:48:38.800004 13346 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41641
I20260620 14:48:38.800097 13346 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:38.800299 13346 heartbeater.cc:511] Master 127.0.98.62:41641 requested a full tablet report, sending...
W20260620 14:48:38.801100 13346 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41641 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:48:38.801162 13346 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:38.801223 13346 heartbeater.cc:511] Master 127.0.98.62:41641 requested a full tablet report, sending...
I20260620 14:48:38.801491 13346 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:38.801561 13346 heartbeater.cc:511] Master 127.0.98.62:41641 requested a full tablet report, sending...
W20260620 14:48:38.801833 13346 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:48:38.904958 13350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:38.905238 13350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:38.905277 13350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:38.906970 13350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:38.907033 13350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:38.907074 13350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:48:38.909256 13350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52463/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36487
--tserver_master_addrs=127.0.98.62:41641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:38.909524 13350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:38.909789 13350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:38.915940 13358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:38.916177 13356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:38.916702 13355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:38.916816 13350 server_base.cc:1034] running on GCE node
I20260620 14:48:38.917037 13350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:38.917321 13350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:38.918486 13350 hybrid_clock.cc:648] HybridClock initialized: now 1781966918918467 us; error 33 us; skew 500 ppm
Jun 20 14:48:38 dist-test-slave-cx89 krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:38.921516 13350 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:48:38.921947 13350 webserver.cc:460] Webserver started at http://127.0.98.2:36487/ using document root <none> and password file <none>
I20260620 14:48:38.922160 13350 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:38.922217 13350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:48:38 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:48:38.939810 13350 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:48:38.942365 13366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:38.942524 13350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:48:38.942603 13350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "a07d805b812547409cba89b947172651"
format_stamp: "Formatted at 2026-06-20 14:48:38 on dist-test-slave-cx89"
server_key: "fee9f2f102ae635234ceabebf4c1fd95"
server_key_iv: "c4b23c003dfe238ffbcfa0acdb3ce90f"
server_key_version: "kuduclusterkey@0"
I20260620 14:48:38.943068 13350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:38.969619 13350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:38.971494 13350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:38.971629 13350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:38.971851 13350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:38.972194 13350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:38.972236 13350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:38.972270 13350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:38.972292 13350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:38.978595 13350 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35623
I20260620 14:48:38.978688 13479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35623 every 8 connection(s)
I20260620 14:48:38.979039 13350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:48:38 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:48:38.986402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 13350
I20260620 14:48:38.986680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52463/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45167
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41641
--webserver_port=38745
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52463/kms with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:48:38.988451 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:38.979812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48127 (local address 127.0.98.62:41641)
0620 14:48:38.979876 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:38.979878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:38.980049 (+ 171us) server_negotiation.cc:407] Connection header received
0620 14:48:38.980579 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:38.980581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:38.980597 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:38.980648 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:38.981173 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.981669 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.982656 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.982872 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.982928 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:38.985229 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:38.985231 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:38.985231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:38.985239 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:38.986812 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:38.987297 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:38.987298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:38.987298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:38.987358 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:38.987667 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:38.987668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:38.987669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:38.987828 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:38.987875 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:38.988261 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 14:48:38.988354 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:48:38.988996 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:38.979794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41641 (local address 127.0.98.2:48127)
0620 14:48:38.979938 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:38.979978 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:38.980487 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:38.980721 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:38.980725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:38.980898 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:38.981090 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:38.981094 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.981755 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.981756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:38.982582 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:38.982584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.982954 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.982956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:38.983091 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:38.983694 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:38.983702 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:38.985128 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:38.986909 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:38.986916 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:38.986919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:38.987201 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:38.987452 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:38.987452 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:38.987453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:38.987586 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:38.987923 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:38.987930 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:38.988106 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 14:48:38.988586 (+ 480us) client_negotiation.cc:240] Negotiation successful
0620 14:48:38.988768 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260620 14:48:38.989437 13480 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41641
I20260620 14:48:38.989537 13480 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:38.989724 13480 heartbeater.cc:511] Master 127.0.98.62:41641 requested a full tablet report, sending...
W20260620 14:48:38.990336 13480 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41641 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:48:38.990415 13480 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:38.990479 13480 heartbeater.cc:511] Master 127.0.98.62:41641 requested a full tablet report, sending...
I20260620 14:48:38.990794 13480 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:38.990875 13480 heartbeater.cc:511] Master 127.0.98.62:41641 requested a full tablet report, sending...
W20260620 14:48:38.991166 13480 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:48:39.094442 13483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:39.094790 13483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:39.094873 13483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:39.096649 13483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:39.096791 13483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:39.096870 13483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:48:39.099074 13483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52463/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38745
--tserver_master_addrs=127.0.98.62:41641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:39.099664 13483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:39.099982 13483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:39.106910 13489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:39.106911 13483 server_base.cc:1034] running on GCE node
W20260620 14:48:39.107429 13490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:39.107636 13492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:39.107885 13483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:39.108141 13483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:39.109302 13483 hybrid_clock.cc:648] HybridClock initialized: now 1781966919109283 us; error 33 us; skew 500 ppm
Jun 20 14:48:39 dist-test-slave-cx89 krb5kdc[12152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:39.115967 13483 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:48:39.116665 13483 webserver.cc:460] Webserver started at http://127.0.98.3:38745/ using document root <none> and password file <none>
I20260620 14:48:39.116938 13483 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:39.117010 13483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:48:39 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:48:39.131484 13483 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:48:39.132378 13500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:39.132570 13483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:39.132674 13483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "47dbe74c36564c6986911b1fae7bf4e6"
format_stamp: "Formatted at 2026-06-20 14:48:38 on dist-test-slave-cx89"
server_key: "64c349ceb622185633b9e6d374b65b7d"
server_key_iv: "8442636652822556326e11e33dcdbc08"
server_key_version: "kuduclusterkey@0"
I20260620 14:48:39.133129 13483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:39.161283 13483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:39.163661 13483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:39.163821 13483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:39.164075 13483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:39.164466 13483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:39.164515 13483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:39.164546 13483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:39.164569 13483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:39.171566 13483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45167
I20260620 14:48:39.172044 13483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:39.172590 13613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45167 every 8 connection(s)
I20260620 14:48:39.175608 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 13483
Jun 20 14:48:39 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:48:39.182700 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:39.172758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44661 (local address 127.0.98.62:41641)
0620 14:48:39.172818 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:39.172820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:39.173065 (+ 245us) server_negotiation.cc:407] Connection header received
0620 14:48:39.173728 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:39.173730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:39.173741 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:39.173807 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:39.174555 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:39.175046 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:39.176581 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:39.176884 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:39.176959 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:39.179483 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:39.179485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:39.179485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:39.179493 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:39.181083 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:39.181568 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:39.181570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:39.181570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:39.181642 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:39.181945 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:39.181946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:39.181947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:39.182089 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:39.182174 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:39.182506 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 14:48:39.182597 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:48:39.182868 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:39.172755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41641 (local address 127.0.98.3:44661)
0620 14:48:39.172912 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:39.172958 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:39.173607 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:39.173931 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:39.173936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:39.174166 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:39.174458 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:39.174465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:39.175142 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:39.175144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:39.176501 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:39.176504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:39.177049 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:39.177050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:39.177195 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:39.177765 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:39.177775 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:39.179331 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:39.181174 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:39.181181 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:39.181184 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:39.181479 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:39.181723 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:39.181724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:39.181725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:39.181872 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:39.182160 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:39.182166 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:39.182357 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 14:48:39.182477 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 14:48:39.182631 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260620 14:48:39.183557 13614 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41641
I20260620 14:48:39.183674 13614 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:39.183881 13614 heartbeater.cc:511] Master 127.0.98.62:41641 requested a full tablet report, sending...
W20260620 14:48:39.184654 13614 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41641 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:48:39.184737 13614 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:39.184813 13614 heartbeater.cc:511] Master 127.0.98.62:41641 requested a full tablet report, sending...
I20260620 14:48:39.185189 13614 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:39.185271 13614 heartbeater.cc:511] Master 127.0.98.62:41641 requested a full tablet report, sending...
W20260620 14:48:39.185588 13614 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:48:39.190073 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:39.176034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50154 (local address 127.0.98.62:41641)
0620 14:48:39.176159 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:39.176160 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:39.176295 (+ 135us) server_negotiation.cc:407] Connection header received
0620 14:48:39.176362 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:39.176363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:39.176376 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:39.176440 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:39.177424 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:39.178020 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:39.178797 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:39.178997 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:39.179056 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:39.182823 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:39.182824 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:39.182825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:39.182832 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:39.184517 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:39.189083 (+ 4566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:39.189085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:39.189085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:39.189173 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:39.189474 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:39.189475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:39.189476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:39.189629 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:39.189823 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:39.189868 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 14:48:39.189968 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":61,"threads_started":1}
Jun 20 14:48:39 dist-test-slave-cx89 krb5kdc[12152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781966919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:39.802016 13346 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:39.802162 13346 heartbeater.cc:511] Master 127.0.98.62:41641 requested a full tablet report, sending...
I20260620 14:48:39.802548 13148 ts_manager.cc:194] Registered new tserver with Master: 12872ae9e3404dc695483e8c0750d2f0 (127.0.98.1:36403)
I20260620 14:48:39.803602 13148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44423
Jun 20 14:48:39 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:48:39.991331 13480 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:39.991487 13480 heartbeater.cc:511] Master 127.0.98.62:41641 requested a full tablet report, sending...
I20260620 14:48:39.991724 13148 ts_manager.cc:194] Registered new tserver with Master: a07d805b812547409cba89b947172651 (127.0.98.2:35623)
I20260620 14:48:39.992249 13148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48127
I20260620 14:48:40.185760 13614 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:40.185916 13614 heartbeater.cc:511] Master 127.0.98.62:41641 requested a full tablet report, sending...
I20260620 14:48:40.186199 13147 ts_manager.cc:194] Registered new tserver with Master: 47dbe74c36564c6986911b1fae7bf4e6 (127.0.98.3:45167)
I20260620 14:48:40.186738 13147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44661
I20260620 14:48:40.197007 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:48:40.205711 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.198041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.0.98.62:41641)
0620 14:48:40.198465 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:40.198467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:40.198473 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:48:40.198496 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:40.198497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:40.198507 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:40.199214 (+ 707us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:40.199252 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.200063 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.201201 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.201504 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.201832 (+ 328us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.202912 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:40.202913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:40.202914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:40.202921 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:40.204521 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:40.204901 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:40.204902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:40.204902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:40.204965 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:40.205162 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:40.205162 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:40.205163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:40.205285 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:40.205382 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:40.205530 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 14:48:40.205612 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":59,"threads_started":1}
I20260620 14:48:40.207736 13147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:48:40.210155 13147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:40.225942 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.217428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36403 (local address 127.0.0.1:53320)
0620 14:48:40.221675 (+ 4247us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:40.221681 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:40.221704 (+ 23us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:40.222048 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:40.222052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:40.222054 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:40.222140 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:40.222142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.223353 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.223354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:40.224656 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:40.224660 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.225635 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.225636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:40.225714 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.225727 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:48:40.225787 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:48:40.225836 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4237,"thread_start_us":62,"threads_started":1}
I20260620 14:48:40.226256 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.217729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.0.98.1:36403)
0620 14:48:40.221792 (+ 4063us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:40.221798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:40.221808 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:48:40.221854 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:40.221856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:40.221872 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:40.221945 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:40.222216 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.223259 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.224753 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.225458 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.225527 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.225603 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:40.226137 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 14:48:40.226180 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4027,"thread_start_us":75,"threads_started":1}
I20260620 14:48:40.226981 13280 tablet_service.cc:1467] Processing CreateTablet for tablet c5cb25a00c1e4c85992fb8db0a3955f4 (DEFAULT_TABLE table=test-table [id=48cf1127ea234ef5a8d8e9355d02606f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:40.227173 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.217215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35623 (local address 127.0.0.1:38046)
0620 14:48:40.221539 (+ 4324us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:40.221550 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:40.221596 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:40.223752 (+ 2156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:40.223754 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:40.223756 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:40.223805 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:40.223807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.225187 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.225188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:40.226177 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:40.226180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.226939 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.226940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:40.226999 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.227003 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:48:40.227055 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:48:40.227099 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4306,"thread_start_us":102,"threads_started":1}
I20260620 14:48:40.227180 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.217740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38046 (local address 127.0.98.2:35623)
0620 14:48:40.223517 (+ 5777us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:40.223522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:40.223534 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:48:40.223568 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:40.223570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:40.223586 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:40.223669 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:40.224106 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.225105 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.226263 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.226851 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.226918 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.226988 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:40.227070 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:48:40.227111 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5738,"thread_start_us":54,"threads_started":1}
I20260620 14:48:40.227365 13280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5cb25a00c1e4c85992fb8db0a3955f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:40.231528 13414 tablet_service.cc:1467] Processing CreateTablet for tablet c5cb25a00c1e4c85992fb8db0a3955f4 (DEFAULT_TABLE table=test-table [id=48cf1127ea234ef5a8d8e9355d02606f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:40.231854 13414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5cb25a00c1e4c85992fb8db0a3955f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:40.235468 13649 tablet_bootstrap.cc:492] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651: Bootstrap starting.
I20260620 14:48:40.236152 13649 tablet_bootstrap.cc:654] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:40.236465 13649 log.cc:824] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:40.239605 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.217195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45167 (local address 127.0.0.1:54062)
0620 14:48:40.219288 (+ 2093us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:40.219349 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:40.223860 (+ 4511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:40.225209 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:40.225210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:40.225212 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:40.225258 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:40.225260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.236850 (+ 11590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.236852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:40.238118 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:40.238123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.239282 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.239283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:40.239362 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.239367 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:48:40.239462 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:48:40.239517 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2058,"thread_start_us":81,"threads_started":1}
I20260620 14:48:40.240132 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.217524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.0.98.3:45167)
0620 14:48:40.217652 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:40.217657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:40.219487 (+ 1830us) server_negotiation.cc:407] Connection header received
0620 14:48:40.223948 (+ 4461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:40.223951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:40.223966 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:40.224025 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:40.225338 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.236736 (+ 11398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.238247 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.239104 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.239170 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.239243 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:40.239994 (+ 751us) server_negotiation.cc:299] Negotiation successful
0620 14:48:40.240050 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":61,"threads_started":1}
I20260620 14:48:40.241008 13548 tablet_service.cc:1467] Processing CreateTablet for tablet c5cb25a00c1e4c85992fb8db0a3955f4 (DEFAULT_TABLE table=test-table [id=48cf1127ea234ef5a8d8e9355d02606f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:40.241346 13548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5cb25a00c1e4c85992fb8db0a3955f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:40.244047 13649 tablet_bootstrap.cc:492] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651: No bootstrap required, opened a new log
I20260620 14:48:40.244133 13649 ts_tablet_manager.cc:1397] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260620 14:48:40.246306 13649 raft_consensus.cc:348] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47dbe74c36564c6986911b1fae7bf4e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45167 } } peers { permanent_uuid: "12872ae9e3404dc695483e8c0750d2f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36403 } } peers { permanent_uuid: "a07d805b812547409cba89b947172651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35623 } }
I20260620 14:48:40.246443 13649 raft_consensus.cc:374] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:40.246474 13649 raft_consensus.cc:729] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07d805b812547409cba89b947172651, State: Initialized, Role: FOLLOWER
I20260620 14:48:40.246629 13649 consensus_queue.cc:260] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47dbe74c36564c6986911b1fae7bf4e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45167 } } peers { permanent_uuid: "12872ae9e3404dc695483e8c0750d2f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36403 } } peers { permanent_uuid: "a07d805b812547409cba89b947172651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35623 } }
I20260620 14:48:40.246881 13649 ts_tablet_manager.cc:1428] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 14:48:40.247000 13480 heartbeater.cc:503] Master 127.0.98.62:41641 was elected leader, sending a full tablet report...
I20260620 14:48:40.247123 13651 tablet_bootstrap.cc:492] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6: Bootstrap starting.
I20260620 14:48:40.248415 13648 tablet_bootstrap.cc:492] T c5cb25a00c1e4c85992fb8db0a3955f4 P 12872ae9e3404dc695483e8c0750d2f0: Bootstrap starting.
I20260620 14:48:40.248469 13651 tablet_bootstrap.cc:654] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:40.248735 13651 log.cc:824] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:40.249061 13648 tablet_bootstrap.cc:654] T c5cb25a00c1e4c85992fb8db0a3955f4 P 12872ae9e3404dc695483e8c0750d2f0: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:40.249349 13648 log.cc:824] T c5cb25a00c1e4c85992fb8db0a3955f4 P 12872ae9e3404dc695483e8c0750d2f0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:40.249583 13651 tablet_bootstrap.cc:492] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6: No bootstrap required, opened a new log
I20260620 14:48:40.249683 13651 ts_tablet_manager.cc:1397] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 14:48:40.250036 13648 tablet_bootstrap.cc:492] T c5cb25a00c1e4c85992fb8db0a3955f4 P 12872ae9e3404dc695483e8c0750d2f0: No bootstrap required, opened a new log
I20260620 14:48:40.250102 13648 ts_tablet_manager.cc:1397] T c5cb25a00c1e4c85992fb8db0a3955f4 P 12872ae9e3404dc695483e8c0750d2f0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:48:40.252290 13651 raft_consensus.cc:348] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47dbe74c36564c6986911b1fae7bf4e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45167 } } peers { permanent_uuid: "12872ae9e3404dc695483e8c0750d2f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36403 } } peers { permanent_uuid: "a07d805b812547409cba89b947172651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35623 } }
I20260620 14:48:40.252606 13648 raft_consensus.cc:348] T c5cb25a00c1e4c85992fb8db0a3955f4 P 12872ae9e3404dc695483e8c0750d2f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47dbe74c36564c6986911b1fae7bf4e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45167 } } peers { permanent_uuid: "12872ae9e3404dc695483e8c0750d2f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36403 } } peers { permanent_uuid: "a07d805b812547409cba89b947172651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35623 } }
I20260620 14:48:40.252763 13648 raft_consensus.cc:374] T c5cb25a00c1e4c85992fb8db0a3955f4 P 12872ae9e3404dc695483e8c0750d2f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:40.252790 13648 raft_consensus.cc:729] T c5cb25a00c1e4c85992fb8db0a3955f4 P 12872ae9e3404dc695483e8c0750d2f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12872ae9e3404dc695483e8c0750d2f0, State: Initialized, Role: FOLLOWER
I20260620 14:48:40.252920 13648 consensus_queue.cc:260] T c5cb25a00c1e4c85992fb8db0a3955f4 P 12872ae9e3404dc695483e8c0750d2f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47dbe74c36564c6986911b1fae7bf4e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45167 } } peers { permanent_uuid: "12872ae9e3404dc695483e8c0750d2f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36403 } } peers { permanent_uuid: "a07d805b812547409cba89b947172651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35623 } }
I20260620 14:48:40.253120 13651 raft_consensus.cc:374] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:40.253157 13648 ts_tablet_manager.cc:1428] T c5cb25a00c1e4c85992fb8db0a3955f4 P 12872ae9e3404dc695483e8c0750d2f0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:40.253223 13651 raft_consensus.cc:729] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47dbe74c36564c6986911b1fae7bf4e6, State: Initialized, Role: FOLLOWER
I20260620 14:48:40.253262 13346 heartbeater.cc:503] Master 127.0.98.62:41641 was elected leader, sending a full tablet report...
I20260620 14:48:40.253410 13651 consensus_queue.cc:260] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47dbe74c36564c6986911b1fae7bf4e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45167 } } peers { permanent_uuid: "12872ae9e3404dc695483e8c0750d2f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36403 } } peers { permanent_uuid: "a07d805b812547409cba89b947172651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35623 } }
I20260620 14:48:40.253733 13651 ts_tablet_manager.cc:1428] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260620 14:48:40.253878 13614 heartbeater.cc:503] Master 127.0.98.62:41641 was elected leader, sending a full tablet report...
Jun 20 14:48:40 dist-test-slave-cx89 krb5kdc[12152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260620 14:48:40.278142 13347 tablet.cc:2369] T c5cb25a00c1e4c85992fb8db0a3955f4 P 12872ae9e3404dc695483e8c0750d2f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:48:40.394505 13652 raft_consensus.cc:482] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:40.394665 13652 raft_consensus.cc:504] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47dbe74c36564c6986911b1fae7bf4e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45167 } } peers { permanent_uuid: "12872ae9e3404dc695483e8c0750d2f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36403 } } peers { permanent_uuid: "a07d805b812547409cba89b947172651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35623 } }
I20260620 14:48:40.395090 13652 leader_election.cc:288] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47dbe74c36564c6986911b1fae7bf4e6 (127.0.98.3:45167), 12872ae9e3404dc695483e8c0750d2f0 (127.0.98.1:36403)
I20260620 14:48:40.407157 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.395707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37127 (local address 127.0.98.3:45167)
0620 14:48:40.395803 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:40.395805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:40.398240 (+ 2435us) server_negotiation.cc:407] Connection header received
0620 14:48:40.398395 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:40.398396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:40.398409 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:40.398438 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:40.398683 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.399387 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.401306 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.401935 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.401986 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.401997 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:40.407003 (+ 5006us) server_negotiation.cc:299] Negotiation successful
0620 14:48:40.407072 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:48:40.407502 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.395297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45167 (local address 127.0.98.2:37127)
0620 14:48:40.398159 (+ 2862us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:40.398173 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:40.398310 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:40.398507 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:40.398507 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:40.398510 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:40.398604 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:40.398606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.399529 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.399530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:40.400963 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:40.400967 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.402082 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.402083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:40.402157 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.402161 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:48:40.407333 (+ 5172us) client_negotiation.cc:240] Negotiation successful
0620 14:48:40.407390 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2841,"thread_start_us":99,"threads_started":1}
I20260620 14:48:40.407836 13568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cb25a00c1e4c85992fb8db0a3955f4" candidate_uuid: "a07d805b812547409cba89b947172651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47dbe74c36564c6986911b1fae7bf4e6" is_pre_election: true
I20260620 14:48:40.407987 13568 raft_consensus.cc:2454] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a07d805b812547409cba89b947172651 in term 0.
I20260620 14:48:40.408255 13367 leader_election.cc:302] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47dbe74c36564c6986911b1fae7bf4e6, a07d805b812547409cba89b947172651; no voters:
I20260620 14:48:40.408389 13652 raft_consensus.cc:2781] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:40.408425 13652 raft_consensus.cc:482] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:40.408444 13652 raft_consensus.cc:3037] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:40.409349 13652 raft_consensus.cc:504] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47dbe74c36564c6986911b1fae7bf4e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45167 } } peers { permanent_uuid: "12872ae9e3404dc695483e8c0750d2f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36403 } } peers { permanent_uuid: "a07d805b812547409cba89b947172651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35623 } }
I20260620 14:48:40.409466 13652 leader_election.cc:288] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [CANDIDATE]: Term 1 election: Requested vote from peers 47dbe74c36564c6986911b1fae7bf4e6 (127.0.98.3:45167), 12872ae9e3404dc695483e8c0750d2f0 (127.0.98.1:36403)
I20260620 14:48:40.409822 13568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cb25a00c1e4c85992fb8db0a3955f4" candidate_uuid: "a07d805b812547409cba89b947172651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47dbe74c36564c6986911b1fae7bf4e6"
I20260620 14:48:40.409901 13568 raft_consensus.cc:3037] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:40.410707 13568 raft_consensus.cc:2454] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a07d805b812547409cba89b947172651 in term 1.
I20260620 14:48:40.410840 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.395520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36403 (local address 127.0.98.2:39011)
0620 14:48:40.407510 (+ 11990us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:40.407525 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:40.407630 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:40.407809 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:40.407810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:40.407812 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:40.407878 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:40.407880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.408717 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.408718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:40.409800 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:40.409802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.410610 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.410611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:40.410677 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.410682 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:48:40.410732 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:48:40.410767 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11963,"thread_start_us":58,"threads_started":1}
I20260620 14:48:40.411041 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.395754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39011 (local address 127.0.98.1:36403)
0620 14:48:40.395811 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:40.395813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:40.407596 (+ 11783us) server_negotiation.cc:407] Connection header received
0620 14:48:40.407703 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:40.407704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:40.407717 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:40.407746 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:40.407962 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.408629 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.409927 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.410482 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.410546 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.410557 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:40.410913 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 14:48:40.410978 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:48:40.411067 13367 leader_election.cc:302] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47dbe74c36564c6986911b1fae7bf4e6, a07d805b812547409cba89b947172651; no voters:
I20260620 14:48:40.411324 13652 raft_consensus.cc:2781] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:40.411357 13300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cb25a00c1e4c85992fb8db0a3955f4" candidate_uuid: "a07d805b812547409cba89b947172651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12872ae9e3404dc695483e8c0750d2f0" is_pre_election: true
I20260620 14:48:40.411509 13652 raft_consensus.cc:686] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [term 1 LEADER]: Becoming Leader. State: Replica: a07d805b812547409cba89b947172651, State: Running, Role: LEADER
I20260620 14:48:40.411559 13300 raft_consensus.cc:2454] T c5cb25a00c1e4c85992fb8db0a3955f4 P 12872ae9e3404dc695483e8c0750d2f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a07d805b812547409cba89b947172651 in term 0.
I20260620 14:48:40.411609 13652 consensus_queue.cc:237] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47dbe74c36564c6986911b1fae7bf4e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45167 } } peers { permanent_uuid: "12872ae9e3404dc695483e8c0750d2f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36403 } } peers { permanent_uuid: "a07d805b812547409cba89b947172651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35623 } }
I20260620 14:48:40.411684 13299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cb25a00c1e4c85992fb8db0a3955f4" candidate_uuid: "a07d805b812547409cba89b947172651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12872ae9e3404dc695483e8c0750d2f0"
I20260620 14:48:40.411731 13299 raft_consensus.cc:3037] T c5cb25a00c1e4c85992fb8db0a3955f4 P 12872ae9e3404dc695483e8c0750d2f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:40.412369 13147 catalog_manager.cc:5495] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 reported cstate change: term changed from 0 to 1, leader changed from <none> to a07d805b812547409cba89b947172651 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a07d805b812547409cba89b947172651" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47dbe74c36564c6986911b1fae7bf4e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12872ae9e3404dc695483e8c0750d2f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a07d805b812547409cba89b947172651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35623 } health_report { overall_health: HEALTHY } } }
I20260620 14:48:40.412516 13299 raft_consensus.cc:2454] T c5cb25a00c1e4c85992fb8db0a3955f4 P 12872ae9e3404dc695483e8c0750d2f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a07d805b812547409cba89b947172651 in term 1.
W20260620 14:48:40.423152 13615 tablet.cc:2369] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:48:40.453542 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.450040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38052 (local address 127.0.98.2:35623)
0620 14:48:40.450081 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:40.450083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:40.450091 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:48:40.450131 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:40.450132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:40.450156 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:40.450217 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:40.450423 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.451151 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.452332 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.452594 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.452657 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.452987 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:40.453124 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:40.453208 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:40.453438 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 14:48:40.453460 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:48:40.458130 13299 raft_consensus.cc:1261] T c5cb25a00c1e4c85992fb8db0a3955f4 P 12872ae9e3404dc695483e8c0750d2f0 [term 1 FOLLOWER]: Refusing update from remote peer a07d805b812547409cba89b947172651: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:40.458581 13652 consensus_queue.cc:1035] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12872ae9e3404dc695483e8c0750d2f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:40.460443 13568 raft_consensus.cc:1261] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6 [term 1 FOLLOWER]: Refusing update from remote peer a07d805b812547409cba89b947172651: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:40.460776 13652 consensus_queue.cc:1035] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47dbe74c36564c6986911b1fae7bf4e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:40.464627 13663 mvcc.cc:205] Tried to move back new op lower bound from 7298936506193334272 to 7298936506006429696. Current Snapshot: MvccSnapshot[applied={T|T < 7298936506193334272}]
I20260620 14:48:40.466998 13147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:40.467089 13147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:40.467945 13147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20e408e734d84575a2f0364dc8f0f1d7: Sending DeleteTablet for 3 replicas of tablet c5cb25a00c1e4c85992fb8db0a3955f4
I20260620 14:48:40.468286 13414 tablet_service.cc:1514] Processing DeleteTablet for tablet c5cb25a00c1e4c85992fb8db0a3955f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38046
I20260620 14:48:40.468690 13548 tablet_service.cc:1514] Processing DeleteTablet for tablet c5cb25a00c1e4c85992fb8db0a3955f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54062
I20260620 14:48:40.468691 13280 tablet_service.cc:1514] Processing DeleteTablet for tablet c5cb25a00c1e4c85992fb8db0a3955f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53320
I20260620 14:48:40.469416 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 13186
I20260620 14:48:40.471859 13670 tablet_replica.cc:330] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651: stopping tablet replica
I20260620 14:48:40.471973 13666 mvcc.cc:205] Tried to move back new op lower bound from 7298936506193334272 to 7298936506006429696. Current Snapshot: MvccSnapshot[applied={T|T < 7298936506193334272}]
I20260620 14:48:40.472007 13670 raft_consensus.cc:2229] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:40.472283 13670 raft_consensus.cc:2258] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:40.472833 13670 ts_tablet_manager.cc:1905] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:40.479540 13672 tablet_replica.cc:330] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6: stopping tablet replica
I20260620 14:48:40.479692 13672 raft_consensus.cc:2229] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:40.479795 13672 raft_consensus.cc:2258] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:40.479868 13670 ts_tablet_manager.cc:1918] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:40.479975 13670 log.cc:1196] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/c5cb25a00c1e4c85992fb8db0a3955f4
I20260620 14:48:40.480391 13670 ts_tablet_manager.cc:1939] T c5cb25a00c1e4c85992fb8db0a3955f4 P a07d805b812547409cba89b947172651: Deleting consensus metadata
I20260620 14:48:40.480918 13133 catalog_manager.cc:4841] TS a07d805b812547409cba89b947172651 (127.0.98.2:35623): tablet c5cb25a00c1e4c85992fb8db0a3955f4 (table test-table [id=48cf1127ea234ef5a8d8e9355d02606f]) successfully deleted
I20260620 14:48:40.481499 13672 ts_tablet_manager.cc:1905] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:48:40.483059 13132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36403 (error 108)
I20260620 14:48:40.483121 13672 ts_tablet_manager.cc:1918] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:40.483187 13672 log.cc:1196] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/c5cb25a00c1e4c85992fb8db0a3955f4
I20260620 14:48:40.483713 13672 ts_tablet_manager.cc:1939] T c5cb25a00c1e4c85992fb8db0a3955f4 P 47dbe74c36564c6986911b1fae7bf4e6: Deleting consensus metadata
I20260620 14:48:40.484176 13132 catalog_manager.cc:4841] TS 47dbe74c36564c6986911b1fae7bf4e6 (127.0.98.3:45167): tablet c5cb25a00c1e4c85992fb8db0a3955f4 (table test-table [id=48cf1127ea234ef5a8d8e9355d02606f]) successfully deleted
I20260620 14:48:40.491556 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 13350
I20260620 14:48:40.495595 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.493165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36403 (local address 127.0.0.1:53330)
0620 14:48:40.495487 (+ 2322us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:40.495550 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2292}
W20260620 14:48:40.495756 13132 catalog_manager.cc:4584] TS 12872ae9e3404dc695483e8c0750d2f0 (127.0.98.1:36403): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c5cb25a00c1e4c85992fb8db0a3955f4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36403: connect: Connection refused (error 111)
I20260620 14:48:40.498692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 13483
I20260620 14:48:40.505762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 13115
I20260620 14:48:40.512850 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:48:40.847005 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:48:40.847090 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:48:41.181172 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:48:41.181 UTC [12182] LOG: received smart shutdown request
2026-06-20 14:48:41.184 UTC [12182] LOG: background worker "logical replication launcher" (PID 12191) exited with exit code 1
2026-06-20 14:48:41.184 UTC [12186] LOG: shutting down
2026-06-20 14:48:41.315 UTC [12182] LOG: database system is shut down
2026-06-20T14:48:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46003 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:41 dist-test-slave-cx89 krb5kdc[13683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:41 dist-test-slave-cx89 krb5kdc[13683](info): set up 2 sockets
Jun 20 14:48:41 dist-test-slave-cx89 krb5kdc[13683](info): commencing operation
krb5kdc: starting...
W20260620 14:48:42.359081 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:48:42 dist-test-slave-cx89 krb5kdc[13683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:48:42.470629 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33301
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33301,127.0.98.61:36321,127.0.98.60:34475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:42.548331 13696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:42.548604 13696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:42.548638 13696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:42.550228 13696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:42.550277 13696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:42.550290 13696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:42.550302 13696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:42.550313 13696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:42.551915 13696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33301,127.0.98.61:36321,127.0.98.60:34475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:42.552151 13696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:42.552358 13696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:42.557040 13702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:42.557091 13704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:42.557091 13701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:42.557135 13696 server_base.cc:1034] running on GCE node
I20260620 14:48:42.557471 13696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:42.557734 13696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:42.558869 13696 hybrid_clock.cc:648] HybridClock initialized: now 1781966922558859 us; error 31 us; skew 500 ppm
Jun 20 14:48:42 dist-test-slave-cx89 krb5kdc[13683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:42.561175 13696 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:48:42.561604 13696 webserver.cc:460] Webserver started at http://127.0.98.62:45183/ using document root <none> and password file <none>
I20260620 14:48:42.561791 13696 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:42.561836 13696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:42.561962 13696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:42.562995 13696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "efb247e1ff054451b774dbac08634c9f"
format_stamp: "Formatted at 2026-06-20 14:48:42 on dist-test-slave-cx89"
server_key: "3d023efdb4df8eecb904b2356e451cac"
server_key_iv: "fb9958d82184e4dbb5165b9328c708d5"
server_key_version: "encryptionkey@0"
I20260620 14:48:42.563318 13696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "efb247e1ff054451b774dbac08634c9f"
format_stamp: "Formatted at 2026-06-20 14:48:42 on dist-test-slave-cx89"
server_key: "3d023efdb4df8eecb904b2356e451cac"
server_key_iv: "fb9958d82184e4dbb5165b9328c708d5"
server_key_version: "encryptionkey@0"
I20260620 14:48:42.564592 13696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:42.565225 13711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:42.565366 13696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:42.565447 13696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "efb247e1ff054451b774dbac08634c9f"
format_stamp: "Formatted at 2026-06-20 14:48:42 on dist-test-slave-cx89"
server_key: "3d023efdb4df8eecb904b2356e451cac"
server_key_iv: "fb9958d82184e4dbb5165b9328c708d5"
server_key_version: "encryptionkey@0"
I20260620 14:48:42.565507 13696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:42.586607 13696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:42.588665 13696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:42.588814 13696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:42.592692 13696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33301
I20260620 14:48:42.592734 13763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33301 every 8 connection(s)
I20260620 14:48:42.593070 13696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:42.593859 13764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:42.594919 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 13696
I20260620 14:48:42.595026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:48:42.595218 392 external_mini_cluster.cc:1428] Setting key 172814d79ef5a4c6932e981f446f3686
I20260620 14:48:42.595642 13764 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:48:42 dist-test-slave-cx89 krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260620 14:48:42.605536 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.596422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33301 (local address 127.0.0.1:49058)
0620 14:48:42.596693 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.596723 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:42.597151 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:42.597418 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:42.597420 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:42.597670 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:42.597821 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.597825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.599220 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.599221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.599972 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.599974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.600609 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.600610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.600659 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.601125 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:42.601133 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:42.602458 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:42.604053 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:42.604059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:42.604061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:42.604296 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:42.604523 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:42.604524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:42.604524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:42.604600 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:42.604971 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:42.604976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:42.605047 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 14:48:42.605126 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:48:42.605353 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":99,"thread_start_us":89,"threads_started":1}
I20260620 14:48:42.605541 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.596437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49058 (local address 127.0.98.62:33301)
0620 14:48:42.596614 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.596624 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.596820 (+ 196us) server_negotiation.cc:407] Connection header received
0620 14:48:42.597290 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.597293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.597315 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.597416 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.597901 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.599127 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.600046 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.600535 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.600648 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.602556 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:42.602563 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:42.602563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:42.602569 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:42.603966 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.604366 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.604368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.604369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.604444 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.604674 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.604675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.604675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.604911 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:42.604983 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:42.605153 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 14:48:42.605300 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260620 14:48:42.606343 13764 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:48:42.606650 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.606549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36321 (local address 127.0.0.1:58230)
0620 14:48:42.606587 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.606629 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 14:48:42.606760 13712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36321: connect: Connection refused (error 111)
I20260620 14:48:42.607101 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.607020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36321 (local address 127.0.0.1:58238)
0620 14:48:42.607055 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.607078 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 14:48:42.607189 13764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36321: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36321: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 14:48:42.617540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36321
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33301,127.0.98.61:36321,127.0.98.60:34475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:48:42.656384 13764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } attempt: 1
I20260620 14:48:42.656857 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.656735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36321 (local address 127.0.0.1:58242)
0620 14:48:42.656788 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.656829 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 14:48:42.657177 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.657099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36321 (local address 127.0.0.1:58244)
0620 14:48:42.657135 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.657157 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 14:48:42.657258 13764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36321: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36321: connect: Connection refused (error 111)
W20260620 14:48:42.695086 13771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:42.695283 13771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:42.695312 13771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:42.696439 13771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:42.696477 13771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:42.696496 13771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:42.696512 13771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:42.696528 13771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:42.697990 13771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33301,127.0.98.61:36321,127.0.98.60:34475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:42.698220 13771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:42.698411 13771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:42.702988 13777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:42.702975 13779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:42.702996 13771 server_base.cc:1034] running on GCE node
W20260620 14:48:42.702965 13776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:42.703457 13771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:42.703728 13771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:42.704905 13771 hybrid_clock.cc:648] HybridClock initialized: now 1781966922704885 us; error 29 us; skew 500 ppm
Jun 20 14:48:42 dist-test-slave-cx89 krb5kdc[13683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:42.707273 13771 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 14:48:42.707723 13771 webserver.cc:460] Webserver started at http://127.0.98.61:37367/ using document root <none> and password file <none>
I20260620 14:48:42.707923 13771 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:42.707973 13771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:42.708098 13771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:42.709131 13771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-1/data/instance:
uuid: "95eea1dd17a14482938b8338b0498d18"
format_stamp: "Formatted at 2026-06-20 14:48:42 on dist-test-slave-cx89"
server_key: "a7a9543722771f6601f873173d040f5a"
server_key_iv: "b5053ee5a6f7ce9314173910d5fcb9b4"
server_key_version: "encryptionkey@0"
I20260620 14:48:42.709486 13771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-1/wal/instance:
uuid: "95eea1dd17a14482938b8338b0498d18"
format_stamp: "Formatted at 2026-06-20 14:48:42 on dist-test-slave-cx89"
server_key: "a7a9543722771f6601f873173d040f5a"
server_key_iv: "b5053ee5a6f7ce9314173910d5fcb9b4"
server_key_version: "encryptionkey@0"
I20260620 14:48:42.710830 13771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:48:42.711514 13786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:42.711649 13771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:48:42.711730 13771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-1/wal
uuid: "95eea1dd17a14482938b8338b0498d18"
format_stamp: "Formatted at 2026-06-20 14:48:42 on dist-test-slave-cx89"
server_key: "a7a9543722771f6601f873173d040f5a"
server_key_iv: "b5053ee5a6f7ce9314173910d5fcb9b4"
server_key_version: "encryptionkey@0"
I20260620 14:48:42.711795 13771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:42.725414 13764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } attempt: 2
I20260620 14:48:42.725813 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.725700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36321 (local address 127.0.0.1:58256)
0620 14:48:42.725755 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.725786 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:48:42.726095 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.726022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36321 (local address 127.0.0.1:58262)
0620 14:48:42.726056 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.726078 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 14:48:42.726164 13764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36321: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36321: connect: Connection refused (error 111)
I20260620 14:48:42.741811 13771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:42.743595 13771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:42.743727 13771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:42.747299 13771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36321
I20260620 14:48:42.747334 13838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36321 every 8 connection(s)
I20260620 14:48:42.747674 13771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:48:42.748479 13839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:42.749922 13839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:48:42.751780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 13771
I20260620 14:48:42.751868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-1/wal/instance
I20260620 14:48:42.752053 392 external_mini_cluster.cc:1428] Setting key 8d837e1d085d354c2bd2593d172e2570
Jun 20 14:48:42 dist-test-slave-cx89 krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 14:48:42.758549 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.750578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49074 (local address 127.0.98.62:33301)
0620 14:48:42.750635 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.750637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.750825 (+ 188us) server_negotiation.cc:407] Connection header received
0620 14:48:42.751268 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.751269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.751281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.751335 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.751949 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.752622 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.753606 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.753810 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.753866 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.756057 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:42.756059 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:42.756059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:42.756067 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:42.757340 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.757702 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.757703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.757703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.757754 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.757967 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.757967 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.757968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.758074 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:42.758127 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:42.758373 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 14:48:42.758463 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:48:42.758694 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.750554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33301 (local address 127.0.0.1:49074)
0620 14:48:42.750714 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.750755 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:42.751192 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:42.751387 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:42.751390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:42.751666 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:42.751854 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.751858 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.752706 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.752706 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.753523 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.753525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.753880 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.753881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.753993 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.754491 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:42.754499 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:42.755972 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:42.757425 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:42.757432 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:42.757434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:42.757637 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:42.757806 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:42.757807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:42.757807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:42.757915 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:42.758127 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:42.758133 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:42.758263 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 14:48:42.758348 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:48:42.758491 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260620 14:48:42.759173 13839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 14:48:42 dist-test-slave-cx89 krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:48:42.768219 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.759393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36321 (local address 127.0.0.1:58278)
0620 14:48:42.759486 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.759498 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:42.759555 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:42.759756 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:42.759757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:42.759861 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:42.759924 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.759925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.761111 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.761112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.761881 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.761884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.762597 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.762598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.762668 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.763064 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:42.763068 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:42.764400 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:42.767080 (+ 2680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:42.767081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:42.767081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:42.767261 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:42.767481 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:42.767481 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:42.767481 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:42.767536 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:42.767788 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:42.767789 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:42.767827 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:48:42.767946 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 14:48:42.768066 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 14:48:42.768232 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.759480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58278 (local address 127.0.98.61:36321)
0620 14:48:42.759637 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.759642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.759659 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:48:42.759679 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.759681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.759691 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.759747 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.759983 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.761034 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.761959 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.762450 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.762526 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.764484 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:42.764492 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:42.764492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:42.764499 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:42.766998 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.767335 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.767339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.767340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.767399 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.767595 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.767595 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.767596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.767727 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:42.767793 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:42.767981 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 14:48:42.768116 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
I20260620 14:48:42.769949 13839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:48:42.770287 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.770179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34475 (local address 127.0.0.1:58302)
0620 14:48:42.770220 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.770263 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 14:48:42.770434 13788 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34475: connect: Connection refused (error 111)
I20260620 14:48:42.770779 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.770705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34475 (local address 127.0.0.1:58306)
0620 14:48:42.770742 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.770764 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 14:48:42.770864 13839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34475: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34475: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 14:48:42.776057 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34475
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33301,127.0.98.61:36321,127.0.98.60:34475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:48:42.817312 13764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } attempt: 3
I20260620 14:48:42.820075 13839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } attempt: 1
I20260620 14:48:42.820448 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.820319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34475 (local address 127.0.0.1:58316)
0620 14:48:42.820375 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.820415 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 14:48:42.820955 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.820872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34475 (local address 127.0.0.1:58320)
0620 14:48:42.820908 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.820933 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 14:48:42.821080 13839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34475: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34475: connect: Connection refused (error 111)
Jun 20 14:48:42 dist-test-slave-cx89 krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:48:42.824245 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.817787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.0.98.61:36321)
0620 14:48:42.817833 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.817835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.817849 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:48:42.817920 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.817921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.817931 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.817997 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.818333 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.818843 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.819882 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.820103 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.820177 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.822014 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:42.822015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:42.822016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:42.822022 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:42.823253 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.823603 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.823604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.823604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.823652 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.823802 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.823803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.823803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.823906 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:42.823959 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:42.824091 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 14:48:42.824157 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:48:42.824251 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.817742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36321 (local address 127.0.0.1:58286)
0620 14:48:42.817805 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.817816 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:42.817859 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:42.818012 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:42.818013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:42.818187 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:42.818266 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.818268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.818915 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.818916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.819782 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.819785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.820194 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.820195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.820272 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.820712 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:42.820718 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:42.821933 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:42.823325 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:42.823326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:42.823327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:42.823536 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:42.823701 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:42.823702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:42.823702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:42.823755 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:42.823961 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:42.823962 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:42.824007 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:48:42.824065 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:48:42.824158 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:48:42.824643 13764 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:48:42.824914 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.824844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34475 (local address 127.0.0.1:58330)
0620 14:48:42.824873 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.824898 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 14:48:42.825177 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.825125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34475 (local address 127.0.0.1:58336)
0620 14:48:42.825147 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.825165 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 14:48:42.825234 13764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34475: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34475: connect: Connection refused (error 111)
W20260620 14:48:42.854768 13846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:42.854987 13846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:42.855016 13846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:42.856153 13846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:42.856192 13846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:42.856211 13846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:42.856226 13846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:42.856246 13846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:42.856367 13764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } attempt: 1
I20260620 14:48:42.856753 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.856644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34475 (local address 127.0.0.1:58348)
0620 14:48:42.856696 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.856726 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 14:48:42.857071 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.856991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34475 (local address 127.0.0.1:58350)
0620 14:48:42.857030 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.857054 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 14:48:42.857142 13764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34475: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34475: connect: Connection refused (error 111)
I20260620 14:48:42.857836 13846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33301,127.0.98.61:36321,127.0.98.60:34475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:42.858069 13846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:42.858253 13846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:42.862629 13854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:42.862650 13851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:42.862650 13852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:42.862887 13846 server_base.cc:1034] running on GCE node
I20260620 14:48:42.863131 13846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:42.863384 13846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:42.864537 13846 hybrid_clock.cc:648] HybridClock initialized: now 1781966922864526 us; error 27 us; skew 500 ppm
Jun 20 14:48:42 dist-test-slave-cx89 krb5kdc[13683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:42.866719 13846 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 14:48:42.867141 13846 webserver.cc:460] Webserver started at http://127.0.98.60:44635/ using document root <none> and password file <none>
I20260620 14:48:42.867331 13846 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:42.867372 13846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:42.867529 13846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:42.868563 13846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-2/data/instance:
uuid: "25abd3cdc01947eca126605dcf7a00ae"
format_stamp: "Formatted at 2026-06-20 14:48:42 on dist-test-slave-cx89"
server_key: "6060b0353514870c257ded01c28b8f1d"
server_key_iv: "5d7afe88d5a5b5376f84353959a5694b"
server_key_version: "encryptionkey@0"
I20260620 14:48:42.868877 13846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-2/wal/instance:
uuid: "25abd3cdc01947eca126605dcf7a00ae"
format_stamp: "Formatted at 2026-06-20 14:48:42 on dist-test-slave-cx89"
server_key: "6060b0353514870c257ded01c28b8f1d"
server_key_iv: "5d7afe88d5a5b5376f84353959a5694b"
server_key_version: "encryptionkey@0"
I20260620 14:48:42.870015 13846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:48:42.870581 13861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:42.870729 13846 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:48:42.870801 13846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-2/wal
uuid: "25abd3cdc01947eca126605dcf7a00ae"
format_stamp: "Formatted at 2026-06-20 14:48:42 on dist-test-slave-cx89"
server_key: "6060b0353514870c257ded01c28b8f1d"
server_key_iv: "5d7afe88d5a5b5376f84353959a5694b"
server_key_version: "encryptionkey@0"
I20260620 14:48:42.870867 13846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:42.884110 13846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:42.885795 13846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:42.885919 13846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:42.889209 13839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } attempt: 2
I20260620 14:48:42.889643 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.889510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34475 (local address 127.0.0.1:58360)
0620 14:48:42.889567 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.889607 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:48:42.889732 13846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34475
I20260620 14:48:42.889861 13913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34475 every 8 connection(s)
I20260620 14:48:42.890152 13846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:48:42.890861 13914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:42.892773 13914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:48:42 dist-test-slave-cx89 krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 14:48:42 dist-test-slave-cx89 krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:48:42.899056 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.889890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34475 (local address 127.0.0.1:58368)
0620 14:48:42.889940 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.889951 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:42.890003 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:42.891492 (+ 1489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:42.891494 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:42.891631 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:42.891674 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.891676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.893180 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.893181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.893857 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.893860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.894478 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.894479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.894524 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.894893 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:42.894897 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:42.896086 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:42.897882 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:42.897883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:42.897883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:42.898059 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:42.898285 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:42.898286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:42.898286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:42.898347 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:42.898743 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:42.898744 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:42.898794 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 14:48:42.898870 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:48:42.898957 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 14:48:42.899200 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.890121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58368 (local address 127.0.98.60:34475)
0620 14:48:42.890615 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.890624 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.890655 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:48:42.891215 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.891224 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.891283 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.891463 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.891770 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.893102 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.893927 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.894405 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.894508 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.896170 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:42.896177 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:42.896178 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:42.896189 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:42.897812 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.898126 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.898128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.898129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.898227 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.898412 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.898412 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.898412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.898684 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:42.898743 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:42.898921 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 14:48:42.899017 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":410,"threads_started":1}
I20260620 14:48:42.900313 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.893430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49086 (local address 127.0.98.62:33301)
0620 14:48:42.893494 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.893496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.893628 (+ 132us) server_negotiation.cc:407] Connection header received
0620 14:48:42.893698 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.893699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.893707 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.893751 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.894208 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.894806 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.895583 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.895775 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.895829 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.897785 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:42.897786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:42.897787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:42.897793 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:42.899155 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.899557 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.899558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.899558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.899613 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.899792 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.899793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.899793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.899920 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:42.899976 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:42.900151 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:48:42.900227 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:48:42.900301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 13846
I20260620 14:48:42.900321 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.893355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33301 (local address 127.0.0.1:49086)
0620 14:48:42.893538 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.893579 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:42.893643 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:42.893783 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:42.893785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:42.894077 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:42.894144 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.894146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.894882 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.894883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.895506 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.895509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.895854 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.895855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.895934 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.896402 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:42.896406 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:42.897708 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:42.899231 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:42.899238 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:42.899240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:42.899489 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:42.899666 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:42.899667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:42.899667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:42.899726 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:42.899984 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:42.899989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:42.900039 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 14:48:42.900120 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:48:42.900231 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260620 14:48:42.900399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966748446763-392-0/minicluster-data/master-2/wal/instance
I20260620 14:48:42.900590 392 external_mini_cluster.cc:1428] Setting key 4a4a9a1f1f3ead260f57c72be8a1a537
I20260620 14:48:42.900861 13839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18: Bootstrap starting.
I20260620 14:48:42.901752 13839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:42.902065 13839 log.cc:824] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:42.902628 13914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:48:42.902948 13839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18: No bootstrap required, opened a new log
I20260620 14:48:42.904657 13839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "95eea1dd17a14482938b8338b0498d18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } } peers { permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } }
I20260620 14:48:42.904764 13839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:42.904795 13839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95eea1dd17a14482938b8338b0498d18, State: Initialized, Role: FOLLOWER
I20260620 14:48:42.904922 13839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "95eea1dd17a14482938b8338b0498d18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } } peers { permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } }
I20260620 14:48:42.905215 13839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:42.905341 13920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "95eea1dd17a14482938b8338b0498d18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } } peers { permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } } }
I20260620 14:48:42.905453 13920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 14:48:42 dist-test-slave-cx89 krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 14:48:42 dist-test-slave-cx89 krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260620 14:48:42.907622 13931 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:48:42.907673 13931 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:48:42.909575 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.901086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58376 (local address 127.0.98.60:34475)
0620 14:48:42.901125 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.901127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.901134 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:48:42.901180 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.901181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.901189 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.901251 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.901760 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.902397 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.903213 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.903496 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.903545 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.905994 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:42.905995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:42.905995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:42.906002 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:42.907997 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.908365 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.908366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.908367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.908436 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.908652 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.908653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.908654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.908777 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:42.908941 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:42.909396 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 14:48:42.909488 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:48:42.907315 13771 master_runner.cc:186] Error getting master registration for 127.0.98.62:33301: OK, instance_id { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" instance_seqno: 1781966922591532 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:48:42.910014 13771 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:48:42.910392 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.902974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58296 (local address 127.0.98.61:36321)
0620 14:48:42.903010 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.903011 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.903018 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:48:42.903037 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.903038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.903047 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.903103 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.903364 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.904048 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.904999 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.905475 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.905538 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.907721 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:42.907722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:42.907722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:42.907729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:42.909276 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.909660 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.909661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.909661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.909713 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.909886 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.909887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.909887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.910044 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:42.910101 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:42.910266 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 14:48:42.910338 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:48:42.910460 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.902889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36321 (local address 127.0.0.1:58296)
0620 14:48:42.902941 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.902952 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:42.902988 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:42.903111 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:42.903112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:42.903251 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:42.903295 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.903297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.904145 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.904146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.904879 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.904881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.905589 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.905590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.905663 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.906226 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:42.906231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:42.907623 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:42.909362 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:42.909363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:42.909364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:42.909585 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:42.909775 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:42.909775 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:42.909776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:42.909829 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:42.910128 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:42.910129 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:42.910168 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:48:42.910235 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:48:42.910319 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:48:42.910956 13914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:48:42 dist-test-slave-cx89 krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 14:48:42 dist-test-slave-cx89 krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:48:42.917659 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.911169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34475 (local address 127.0.0.1:58378)
0620 14:48:42.911204 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.911212 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:42.911244 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:42.911390 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:42.911391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:42.911561 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:42.911596 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.911598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.912394 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.912395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.912856 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.912857 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.913157 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.913157 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.913200 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.913535 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:42.913539 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:42.914875 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:42.916692 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:42.916693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:42.916693 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:42.916877 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:42.917073 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:42.917073 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:42.917073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:42.917121 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:42.917358 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:42.917359 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:42.917406 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:48:42.917464 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:48:42.917558 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 14:48:42.917883 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.911209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58378 (local address 127.0.98.60:34475)
0620 14:48:42.911298 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.911300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.911305 (+ 5us) server_negotiation.cc:407] Connection header received
0620 14:48:42.911319 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.911320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.911327 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.911371 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.911673 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.912334 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.912923 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.913107 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.913154 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.915397 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:42.915398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:42.915398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:42.915405 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:42.916624 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.916937 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.916946 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.916946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.917005 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.917175 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.917176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.917177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.917303 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:42.917359 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:42.917489 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 14:48:42.917602 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":77}
I20260620 14:48:42.918977 13914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae: Bootstrap starting.
I20260620 14:48:42.919623 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.910829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.0.98.62:33301)
0620 14:48:42.910864 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.910865 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.910873 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:48:42.910932 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.910933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.910942 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.910985 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.911557 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.912261 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.913227 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.913512 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.913574 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.915859 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:42.915860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:42.915861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:42.915867 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:42.917755 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.918137 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.918138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.918139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.918193 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.918363 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.918364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.918364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.918484 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:42.919044 (+ 560us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:42.919441 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 14:48:42.919533 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:48:42.919956 13914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:42.920202 13914 log.cc:824] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:42.921001 13914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae: No bootstrap required, opened a new log
I20260620 14:48:42.922930 13914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "95eea1dd17a14482938b8338b0498d18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } } peers { permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } }
I20260620 14:48:42.923092 13914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:42.923125 13914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25abd3cdc01947eca126605dcf7a00ae, State: Initialized, Role: FOLLOWER
I20260620 14:48:42.923242 13914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "95eea1dd17a14482938b8338b0498d18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } } peers { permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } }
I20260620 14:48:42.923573 13914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:42.923540 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "95eea1dd17a14482938b8338b0498d18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } } peers { permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } } }
I20260620 14:48:42.923732 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 14:48:42 dist-test-slave-cx89 krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260620 14:48:42.925607 13945 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:48:42.925695 13945 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:48:42.928117 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.920876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58308 (local address 127.0.98.61:36321)
0620 14:48:42.920909 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.920911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.920919 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:48:42.920970 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.920971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.920980 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.921024 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.921450 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.921912 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.922523 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.922715 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.922765 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.925083 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:42.925084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:42.925085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:42.925092 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:42.926605 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.927013 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.927014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.927015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.927071 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.927249 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.927249 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.927250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.927369 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:42.927549 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:42.927958 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 14:48:42.928032 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:48:42.925635 13846 master_runner.cc:186] Error getting master registration for 127.0.98.62:33301: OK, instance_id { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" instance_seqno: 1781966922591532 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:48:42.928192 13846 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:48:42.929523 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:48:42.932264 13764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } attempt: 2
Jun 20 14:48:42 dist-test-slave-cx89 krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:48:42.938014 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.932489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34475 (local address 127.0.0.1:58392)
0620 14:48:42.932533 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.932542 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:42.932577 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:42.932684 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:42.932684 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:42.932820 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:42.932857 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.932859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.933427 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.933428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.933913 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.933915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.934216 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.934216 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.934269 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.934634 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:42.934639 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:42.935773 (+ 1134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:42.937141 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:42.937142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:42.937142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:42.937298 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:42.937462 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:42.937462 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:42.937463 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:42.937519 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:42.937723 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:42.937723 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:42.937769 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 14:48:42.937828 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:48:42.937920 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 14:48:42.938014 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.932563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58392 (local address 127.0.98.60:34475)
0620 14:48:42.932603 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.932604 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.932611 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:48:42.932630 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.932631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.932639 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.932682 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.932916 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.933364 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.933978 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.934163 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.934212 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.935849 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:42.935850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:42.935851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:42.935857 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:42.937067 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.937356 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.937357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.937358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.937410 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.937566 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.937566 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.937567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.937671 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:42.937725 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:42.937853 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 14:48:42.937924 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:48:42.939558 13764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f: Bootstrap starting.
I20260620 14:48:42.940335 13764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:42.940621 13764 log.cc:824] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:42.944720 13764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f: No bootstrap required, opened a new log
I20260620 14:48:42.947160 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.930945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58314 (local address 127.0.98.61:36321)
0620 14:48:42.930987 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.930988 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.940032 (+ 9044us) server_negotiation.cc:407] Connection header received
0620 14:48:42.940113 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.940114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.940124 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.940150 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.940868 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.941561 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.942546 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.942825 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.942862 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.944534 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:42.944536 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:42.944536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:42.944543 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:42.945992 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.946359 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.946360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.946361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.946426 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.946617 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.946617 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.946618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.946736 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:42.946803 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:42.946997 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 14:48:42.947071 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 14:48:42.947314 13764 raft_consensus.cc:348] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "95eea1dd17a14482938b8338b0498d18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } } peers { permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } }
I20260620 14:48:42.947465 13764 raft_consensus.cc:374] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:42.947503 13764 raft_consensus.cc:729] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb247e1ff054451b774dbac08634c9f, State: Initialized, Role: FOLLOWER
I20260620 14:48:42.947625 13764 consensus_queue.cc:260] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "95eea1dd17a14482938b8338b0498d18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } } peers { permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } }
I20260620 14:48:42.947973 13764 sys_catalog.cc:564] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:42.951545 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "95eea1dd17a14482938b8338b0498d18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } } peers { permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } } }
I20260620 14:48:42.951642 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:48:42.952356 13966 catalog_manager.cc:1558] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:48:42.952407 13966 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:48:42.952613 13696 master_runner.cc:263] No leader master found from master efb247e1ff054451b774dbac08634c9f
I20260620 14:48:42.952656 13696 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:48:42.954402 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.940968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58408 (local address 127.0.98.60:34475)
0620 14:48:42.941071 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.941073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.947504 (+ 6431us) server_negotiation.cc:407] Connection header received
0620 14:48:42.947592 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.947593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.947603 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.947629 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.948123 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.948811 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.949677 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.949933 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.949972 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.951183 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:42.951185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:42.951185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:42.951191 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:42.952663 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.953021 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.953022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.953022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.953076 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.953259 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.953260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.953261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.953377 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:42.953472 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:42.954020 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 14:48:42.954300 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 14:48:42.955884 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.930753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49098 (local address 127.0.98.62:33301)
0620 14:48:42.930795 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.930797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.940835 (+ 10038us) server_negotiation.cc:407] Connection header received
0620 14:48:42.940855 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.940856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.940865 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.940922 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.941675 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.942372 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.943256 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.943554 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.943595 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.953362 (+ 9767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:42.953363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:42.953363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:42.953370 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:42.954783 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.955116 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.955117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.955117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.955172 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:42.955370 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:42.955371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:42.955371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:42.955532 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:42.955612 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:42.955737 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:48:42.955801 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:48:42.969338 13920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:42.969466 13920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "95eea1dd17a14482938b8338b0498d18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } } peers { permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } }
I20260620 14:48:42.969703 13920 leader_election.cc:288] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efb247e1ff054451b774dbac08634c9f (127.0.98.62:33301), 25abd3cdc01947eca126605dcf7a00ae (127.0.98.60:34475)
I20260620 14:48:42.970003 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95eea1dd17a14482938b8338b0498d18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb247e1ff054451b774dbac08634c9f" is_pre_election: true
I20260620 14:48:42.970149 13738 raft_consensus.cc:2454] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95eea1dd17a14482938b8338b0498d18 in term 0.
I20260620 14:48:42.970322 13787 leader_election.cc:302] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95eea1dd17a14482938b8338b0498d18, efb247e1ff054451b774dbac08634c9f; no voters:
I20260620 14:48:42.970806 13888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95eea1dd17a14482938b8338b0498d18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25abd3cdc01947eca126605dcf7a00ae" is_pre_election: true
I20260620 14:48:42.970957 13888 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95eea1dd17a14482938b8338b0498d18 in term 0.
I20260620 14:48:42.971231 13920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:42.971283 13920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:42.971305 13920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:42.972275 13920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "95eea1dd17a14482938b8338b0498d18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } } peers { permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } }
I20260620 14:48:42.972390 13920 leader_election.cc:288] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [CANDIDATE]: Term 1 election: Requested vote from peers efb247e1ff054451b774dbac08634c9f (127.0.98.62:33301), 25abd3cdc01947eca126605dcf7a00ae (127.0.98.60:34475)
I20260620 14:48:42.972609 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95eea1dd17a14482938b8338b0498d18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb247e1ff054451b774dbac08634c9f"
I20260620 14:48:42.972674 13738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:42.973534 13738 raft_consensus.cc:2454] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95eea1dd17a14482938b8338b0498d18 in term 1.
I20260620 14:48:42.973728 13787 leader_election.cc:302] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95eea1dd17a14482938b8338b0498d18, efb247e1ff054451b774dbac08634c9f; no voters:
I20260620 14:48:42.973976 13888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95eea1dd17a14482938b8338b0498d18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25abd3cdc01947eca126605dcf7a00ae"
I20260620 14:48:42.974064 13888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:42.974835 13888 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95eea1dd17a14482938b8338b0498d18 in term 1.
I20260620 14:48:42.975077 13920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:42.975250 13920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [term 1 LEADER]: Becoming Leader. State: Replica: 95eea1dd17a14482938b8338b0498d18, State: Running, Role: LEADER
I20260620 14:48:42.975332 13920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "95eea1dd17a14482938b8338b0498d18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } } peers { permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } }
I20260620 14:48:42.975703 13920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95eea1dd17a14482938b8338b0498d18. Latest consensus state: current_term: 1 leader_uuid: "95eea1dd17a14482938b8338b0498d18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "95eea1dd17a14482938b8338b0498d18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } } peers { permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } } }
I20260620 14:48:42.975766 13920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:42.976788 13970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:42.977044 13970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:42.978994 13888 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae [term 1 FOLLOWER]: Refusing update from remote peer 95eea1dd17a14482938b8338b0498d18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:42.979336 13920 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:42.980760 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95eea1dd17a14482938b8338b0498d18. Latest consensus state: current_term: 1 leader_uuid: "95eea1dd17a14482938b8338b0498d18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "95eea1dd17a14482938b8338b0498d18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } } peers { permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } } }
I20260620 14:48:42.980839 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:48:42.981256 13738 raft_consensus.cc:1261] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f [term 1 FOLLOWER]: Refusing update from remote peer 95eea1dd17a14482938b8338b0498d18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:42.981506 13920 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:42.982601 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95eea1dd17a14482938b8338b0498d18. Latest consensus state: current_term: 1 leader_uuid: "95eea1dd17a14482938b8338b0498d18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "95eea1dd17a14482938b8338b0498d18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } } peers { permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } } }
I20260620 14:48:42.982681 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:48:42.983695 13968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95eea1dd17a14482938b8338b0498d18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "95eea1dd17a14482938b8338b0498d18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } } peers { permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } } }
I20260620 14:48:42.983789 13968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:42.984221 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95eea1dd17a14482938b8338b0498d18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "95eea1dd17a14482938b8338b0498d18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } } peers { permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } } }
I20260620 14:48:42.984298 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:48:42.984521 13971 mvcc.cc:205] Tried to move back new op lower bound from 7298936516519837696 to 7298936516507549696. Current Snapshot: MvccSnapshot[applied={T|T < 7298936516519837696}]
I20260620 14:48:42.985858 13972 mvcc.cc:205] Tried to move back new op lower bound from 7298936516519837696 to 7298936516507549696. Current Snapshot: MvccSnapshot[applied={T|T < 7298936516519837696}]
I20260620 14:48:42.986392 13970 catalog_manager.cc:1347] Generated new cluster ID: 3185f9d193a3443d8ccc265f4e276b24
I20260620 14:48:42.986434 13970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:42.988426 13920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95eea1dd17a14482938b8338b0498d18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "95eea1dd17a14482938b8338b0498d18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } } peers { permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } } }
I20260620 14:48:42.988503 13920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:42.988881 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95eea1dd17a14482938b8338b0498d18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb247e1ff054451b774dbac08634c9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "95eea1dd17a14482938b8338b0498d18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36321 } } peers { permanent_uuid: "25abd3cdc01947eca126605dcf7a00ae" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34475 } } }
I20260620 14:48:42.988952 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:48:42.989152 13974 mvcc.cc:205] Tried to move back new op lower bound from 7298936516519837696 to 7298936516507549696. Current Snapshot: MvccSnapshot[applied={T|T < 7298936516519837696}]
I20260620 14:48:43.047230 13970 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:43.049008 13970 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:43.062351 13970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95eea1dd17a14482938b8338b0498d18: Generated new TSK 0
I20260620 14:48:43.063529 13970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:48:43.926137 13945 catalog_manager.cc:1259] Loaded cluster ID: 3185f9d193a3443d8ccc265f4e276b24
I20260620 14:48:43.926209 13945 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae: loading cluster ID for follower catalog manager: success
I20260620 14:48:43.927001 13945 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae: acquiring CA information for follower catalog manager: success
I20260620 14:48:43.927289 13945 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 25abd3cdc01947eca126605dcf7a00ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:48:43.952833 13966 catalog_manager.cc:1259] Loaded cluster ID: 3185f9d193a3443d8ccc265f4e276b24
I20260620 14:48:43.952904 13966 catalog_manager.cc:1552] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f: loading cluster ID for follower catalog manager: success
I20260620 14:48:43.953658 13966 catalog_manager.cc:1574] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f: acquiring CA information for follower catalog manager: success
I20260620 14:48:43.953922 13966 catalog_manager.cc:1602] T 00000000000000000000000000000000 P efb247e1ff054451b774dbac08634c9f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:48:44.124686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 13696
I20260620 14:48:44.128947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 13771
I20260620 14:48:44.132913 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 13846
2026-06-20T14:48:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2730 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:44 dist-test-slave-cx89 krb5kdc[13987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:44 dist-test-slave-cx89 krb5kdc[13987](info): set up 2 sockets
Jun 20 14:48:44 dist-test-slave-cx89 krb5kdc[13987](info): commencing operation
krb5kdc: starting...
W20260620 14:48:45.091094 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:48:45 dist-test-slave-cx89 krb5kdc[13987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:48:45.199818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45355
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:45.278924 14000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:45.279134 14000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:45.279156 14000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:45.280306 14000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:45.280347 14000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:45.280359 14000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:45.280370 14000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:45.280385 14000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:45.281764 14000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:45.281944 14000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:45.282116 14000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:45.286520 14006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:45.286657 14000 server_base.cc:1034] running on GCE node
W20260620 14:48:45.286571 14008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:45.286537 14005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:45.286929 14000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:45.287171 14000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:45.288316 14000 hybrid_clock.cc:648] HybridClock initialized: now 1781966925288299 us; error 32 us; skew 500 ppm
Jun 20 14:48:45 dist-test-slave-cx89 krb5kdc[13987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:45.290596 14000 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:48:45.290999 14000 webserver.cc:460] Webserver started at http://127.0.98.62:33437/ using document root <none> and password file <none>
I20260620 14:48:45.291193 14000 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:45.291234 14000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:45.291352 14000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:45.292348 14000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "0dba037d02a34db4a668f4f41f769d2e"
format_stamp: "Formatted at 2026-06-20 14:48:45 on dist-test-slave-cx89"
server_key: "194b99f2e955fd032e3e726f03061afa"
server_key_iv: "cd4fb21fd3c0d2c1603228d1e6eb0ebb"
server_key_version: "encryptionkey@0"
I20260620 14:48:45.292658 14000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "0dba037d02a34db4a668f4f41f769d2e"
format_stamp: "Formatted at 2026-06-20 14:48:45 on dist-test-slave-cx89"
server_key: "194b99f2e955fd032e3e726f03061afa"
server_key_iv: "cd4fb21fd3c0d2c1603228d1e6eb0ebb"
server_key_version: "encryptionkey@0"
I20260620 14:48:45.293869 14000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:48:45.294512 14015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:45.294652 14000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:48:45.294732 14000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "0dba037d02a34db4a668f4f41f769d2e"
format_stamp: "Formatted at 2026-06-20 14:48:45 on dist-test-slave-cx89"
server_key: "194b99f2e955fd032e3e726f03061afa"
server_key_iv: "cd4fb21fd3c0d2c1603228d1e6eb0ebb"
server_key_version: "encryptionkey@0"
I20260620 14:48:45.294788 14000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:45.318745 14000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:45.320708 14000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:45.320839 14000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:45.324612 14000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45355
I20260620 14:48:45.324644 14067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45355 every 8 connection(s)
I20260620 14:48:45.324972 14000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:45.325762 14068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:45.328038 14068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e: Bootstrap starting.
I20260620 14:48:45.328585 14068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:45.328835 14068 log.cc:824] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:45.329463 14068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e: No bootstrap required, opened a new log
I20260620 14:48:45.330987 14068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dba037d02a34db4a668f4f41f769d2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45355 } }
I20260620 14:48:45.331127 14068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:45.331151 14068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dba037d02a34db4a668f4f41f769d2e, State: Initialized, Role: FOLLOWER
I20260620 14:48:45.331276 14068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dba037d02a34db4a668f4f41f769d2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45355 } }
I20260620 14:48:45.331343 14068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:45.331363 14068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:45.331390 14068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:45.331907 14068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dba037d02a34db4a668f4f41f769d2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45355 } }
I20260620 14:48:45.331991 14068 leader_election.cc:302] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dba037d02a34db4a668f4f41f769d2e; no voters:
I20260620 14:48:45.332185 14068 leader_election.cc:288] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:45.332211 14071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:45.332343 14071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e [term 1 LEADER]: Becoming Leader. State: Replica: 0dba037d02a34db4a668f4f41f769d2e, State: Running, Role: LEADER
I20260620 14:48:45.332481 14068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:45.332446 14071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dba037d02a34db4a668f4f41f769d2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45355 } }
I20260620 14:48:45.332835 14072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dba037d02a34db4a668f4f41f769d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dba037d02a34db4a668f4f41f769d2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45355 } } }
I20260620 14:48:45.332909 14072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:45.332974 14073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dba037d02a34db4a668f4f41f769d2e. Latest consensus state: current_term: 1 leader_uuid: "0dba037d02a34db4a668f4f41f769d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dba037d02a34db4a668f4f41f769d2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45355 } } }
I20260620 14:48:45.333048 14073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:45.333189 14078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:45.333698 14078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:45.334259 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 14000
I20260620 14:48:45.334357 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:48:45.334550 392 external_mini_cluster.cc:1428] Setting key 3361b3d8c37fd72904145845292c30d0
I20260620 14:48:45.335392 14078 catalog_manager.cc:1347] Generated new cluster ID: 9ed71d74f87a42cfaeca32b04d1f1ac6
I20260620 14:48:45.335455 14078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:48:45 dist-test-slave-cx89 krb5kdc[13987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:48:45.345741 14091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:45.334948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35474 (local address 127.0.98.62:45355)
0620 14:48:45.335469 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:45.335478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:45.335509 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:48:45.335621 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:45.335628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:45.335681 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:45.335816 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:45.336504 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:45.337929 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:45.338897 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:45.339538 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:45.339662 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:45.342301 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:45.342305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:45.342305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:45.342317 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:45.344173 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:45.344561 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:45.344563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:45.344564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:45.344679 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:45.344867 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:45.344867 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:45.344868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:45.345157 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:45.345276 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:45.345399 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 14:48:45.345531 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":59,"threads_started":1}
I20260620 14:48:45.346061 14078 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:45.346678 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:48:45.346755 14078 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:45.374627 14078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dba037d02a34db4a668f4f41f769d2e: Generated new TSK 0
I20260620 14:48:45.374841 14078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:48:45.400106 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 14000
2026-06-20T14:48:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1264 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:45 dist-test-slave-cx89 krb5kdc[14095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:45 dist-test-slave-cx89 krb5kdc[14095](info): set up 2 sockets
Jun 20 14:48:45 dist-test-slave-cx89 krb5kdc[14095](info): commencing operation
krb5kdc: starting...
W20260620 14:48:46.343386 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:48:46 dist-test-slave-cx89 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:48:46.451578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41253
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:46.528625 14108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:46.528867 14108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:46.528895 14108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:46.530017 14108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:46.530057 14108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:46.530074 14108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:46.530092 14108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:46.530107 14108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:46.531554 14108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:46.531785 14108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:46.531975 14108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:46.536375 14113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:46.536464 14108 server_base.cc:1034] running on GCE node
W20260620 14:48:46.536398 14114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:46.536434 14116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:46.536761 14108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:46.537046 14108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:46.538180 14108 hybrid_clock.cc:648] HybridClock initialized: now 1781966926538170 us; error 28 us; skew 500 ppm
Jun 20 14:48:46 dist-test-slave-cx89 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:46.540467 14108 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:48:46.540843 14108 webserver.cc:460] Webserver started at http://127.0.98.62:44415/ using document root <none> and password file <none>
I20260620 14:48:46.541025 14108 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:46.541081 14108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:46.541199 14108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:46.542238 14108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "09c65c0552ec45ccb4901fd5f5283b07"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-cx89"
server_key: "da08776fe610d22b660a847f6ae402fc"
server_key_iv: "b54af5fc5278cee6b60594db91788a57"
server_key_version: "encryptionkey@0"
I20260620 14:48:46.542564 14108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "09c65c0552ec45ccb4901fd5f5283b07"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-cx89"
server_key: "da08776fe610d22b660a847f6ae402fc"
server_key_iv: "b54af5fc5278cee6b60594db91788a57"
server_key_version: "encryptionkey@0"
I20260620 14:48:46.543803 14108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:48:46.544410 14123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:46.544567 14108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:48:46.544644 14108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "09c65c0552ec45ccb4901fd5f5283b07"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-cx89"
server_key: "da08776fe610d22b660a847f6ae402fc"
server_key_iv: "b54af5fc5278cee6b60594db91788a57"
server_key_version: "encryptionkey@0"
I20260620 14:48:46.544701 14108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:46.556255 14108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:46.558039 14108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:46.558174 14108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:46.562271 14108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41253
I20260620 14:48:46.562307 14175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41253 every 8 connection(s)
I20260620 14:48:46.562738 14108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:46.563874 14176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:46.565579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 14108
I20260620 14:48:46.565667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:48:46.565842 392 external_mini_cluster.cc:1428] Setting key f0225d45cc3af8014c20ae5540ce28d6
I20260620 14:48:46.566310 14176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07: Bootstrap starting.
I20260620 14:48:46.567004 14176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:46.567309 14176 log.cc:824] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:46.568023 14176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07: No bootstrap required, opened a new log
I20260620 14:48:46.570137 14176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c65c0552ec45ccb4901fd5f5283b07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260620 14:48:46.570299 14176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:46.570330 14176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c65c0552ec45ccb4901fd5f5283b07, State: Initialized, Role: FOLLOWER
I20260620 14:48:46.570448 14176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c65c0552ec45ccb4901fd5f5283b07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260620 14:48:46.570519 14176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:46.570554 14176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:46.570596 14176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:46.571161 14176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c65c0552ec45ccb4901fd5f5283b07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260620 14:48:46.571249 14176 leader_election.cc:302] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09c65c0552ec45ccb4901fd5f5283b07; no voters:
I20260620 14:48:46.571404 14176 leader_election.cc:288] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:46.571478 14181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:46.571588 14181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 1 LEADER]: Becoming Leader. State: Replica: 09c65c0552ec45ccb4901fd5f5283b07, State: Running, Role: LEADER
I20260620 14:48:46.571678 14176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:46.571722 14181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c65c0552ec45ccb4901fd5f5283b07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260620 14:48:46.572086 14181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09c65c0552ec45ccb4901fd5f5283b07. Latest consensus state: current_term: 1 leader_uuid: "09c65c0552ec45ccb4901fd5f5283b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c65c0552ec45ccb4901fd5f5283b07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } } }
I20260620 14:48:46.572297 14181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:46.572237 14183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09c65c0552ec45ccb4901fd5f5283b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c65c0552ec45ccb4901fd5f5283b07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } } }
I20260620 14:48:46.572652 14183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [sys.catalog]: This master's current role is: LEADER
Jun 20 14:48:46 dist-test-slave-cx89 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:48:46.572937 14190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:46.573585 14190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:46.575398 14190 catalog_manager.cc:1347] Generated new cluster ID: 9cdefb78af174a258722df1610dd9cf6
I20260620 14:48:46.575467 14190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:46.576747 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:46.566308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43332 (local address 127.0.98.62:41253)
0620 14:48:46.566488 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:46.566498 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:46.566528 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:48:46.567090 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:46.567099 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:46.567171 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:46.567326 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:46.567944 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.569294 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.570027 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.570622 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.570734 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:46.573363 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:46.573367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:46.573368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:46.573379 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:46.575095 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:46.575548 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:46.575551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:46.575552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:46.575651 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:46.575860 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:46.575860 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:46.575861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:46.576140 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:46.576267 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:46.576392 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:48:46.576536 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260620 14:48:46.587309 14190 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:46.588080 14190 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:46.596678 14190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07: Generated new TSK 0
I20260620 14:48:46.596872 14190 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:48:46.622032 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41253
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:46.700280 14202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:46.700480 14202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:46.700501 14202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:46.701596 14202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:46.701630 14202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:46.701653 14202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:48:46.703055 14202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:46.703245 14202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:46.703444 14202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:46.708086 14210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:46.708084 14207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:46.708091 14208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:46.708351 14202 server_base.cc:1034] running on GCE node
I20260620 14:48:46.708523 14202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:46.708734 14202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:46.709863 14202 hybrid_clock.cc:648] HybridClock initialized: now 1781966926709849 us; error 32 us; skew 500 ppm
Jun 20 14:48:46 dist-test-slave-cx89 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:46.712332 14202 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:48:46.712728 14202 webserver.cc:460] Webserver started at http://127.0.98.1:41889/ using document root <none> and password file <none>
I20260620 14:48:46.712934 14202 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:46.712980 14202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:46.713099 14202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:46.714076 14202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "11b228dfaee14cb484bb91c91a417a52"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-cx89"
server_key: "212c10ccd862fc0099bbd347e8e82128"
server_key_iv: "5b639541e85e8882f3c7f63427a82298"
server_key_version: "encryptionkey@0"
I20260620 14:48:46.714406 14202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "11b228dfaee14cb484bb91c91a417a52"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-cx89"
server_key: "212c10ccd862fc0099bbd347e8e82128"
server_key_iv: "5b639541e85e8882f3c7f63427a82298"
server_key_version: "encryptionkey@0"
I20260620 14:48:46.715636 14202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:48:46.716302 14217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:46.716434 14202 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:48:46.716513 14202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "11b228dfaee14cb484bb91c91a417a52"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-cx89"
server_key: "212c10ccd862fc0099bbd347e8e82128"
server_key_iv: "5b639541e85e8882f3c7f63427a82298"
server_key_version: "encryptionkey@0"
I20260620 14:48:46.716578 14202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:46.732410 14202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:46.734493 14202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:46.734624 14202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:46.734825 14202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:46.735119 14202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:46.735159 14202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:46.735188 14202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:46.735204 14202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:46.740469 14202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39717
I20260620 14:48:46.740536 14330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39717 every 8 connection(s)
I20260620 14:48:46.740809 14202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:46.746291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 14202
I20260620 14:48:46.746383 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
Jun 20 14:48:46 dist-test-slave-cx89 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:48:46.746579 392 external_mini_cluster.cc:1428] Setting key 0b063ae6f248d62ab391f96dc2c20b02
I20260620 14:48:46.749536 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:46.741480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48957 (local address 127.0.98.62:41253)
0620 14:48:46.741529 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:46.741531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:46.741741 (+ 210us) server_negotiation.cc:407] Connection header received
0620 14:48:46.742193 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:46.742193 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:46.742203 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:46.742252 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:46.743132 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.743653 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.744659 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.744906 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.744968 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:46.746927 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:46.746929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:46.746930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:46.746938 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:46.748292 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:46.748664 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:46.748665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:46.748665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:46.748722 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:46.748939 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:46.748940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:46.748940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:46.749050 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:46.749103 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:46.749361 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 14:48:46.749446 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:48:46.749708 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:46.741465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:48957)
0620 14:48:46.741626 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:46.741662 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:46.742121 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:46.742315 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:46.742318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:46.742876 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:46.743065 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:46.743069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.743725 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.743726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:46.744588 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:46.744591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.744978 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.744979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:46.745073 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:46.745505 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:46.745512 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:46.746839 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:46.748360 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:46.748369 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:46.748371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:46.748596 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:46.748776 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:46.748776 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:46.748777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:46.748883 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:46.749104 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:46.749110 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:46.749248 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 14:48:46.749337 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:48:46.749488 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260620 14:48:46.750176 14331 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41253
I20260620 14:48:46.750277 14331 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:46.750453 14331 heartbeater.cc:511] Master 127.0.98.62:41253 requested a full tablet report, sending...
I20260620 14:48:46.750999 14140 ts_manager.cc:194] Registered new tserver with Master: 11b228dfaee14cb484bb91c91a417a52 (127.0.98.1:39717)
I20260620 14:48:46.751637 14140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48957
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:48:46.769833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41253
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:46.847522 14336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:46.847713 14336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:46.847735 14336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:46.848855 14336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:46.848893 14336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:46.848919 14336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:48:46.850334 14336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:46.850541 14336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:46.850747 14336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:46.855474 14341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:46.855599 14336 server_base.cc:1034] running on GCE node
W20260620 14:48:46.855580 14342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:46.855481 14344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:46.855886 14336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:46.856117 14336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:46.857252 14336 hybrid_clock.cc:648] HybridClock initialized: now 1781966926857241 us; error 29 us; skew 500 ppm
Jun 20 14:48:46 dist-test-slave-cx89 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:46.859560 14336 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:48:46.859989 14336 webserver.cc:460] Webserver started at http://127.0.98.2:42193/ using document root <none> and password file <none>
I20260620 14:48:46.860186 14336 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:46.860231 14336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:46.860355 14336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:46.861367 14336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "7c0d5fdb8bce49d994469114de53dd14"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-cx89"
server_key: "48e2afd9fd9fde61317ff9650154caab"
server_key_iv: "152bf0a8c8c646dfbca3e4f6806b2441"
server_key_version: "encryptionkey@0"
I20260620 14:48:46.861717 14336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7c0d5fdb8bce49d994469114de53dd14"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-cx89"
server_key: "48e2afd9fd9fde61317ff9650154caab"
server_key_iv: "152bf0a8c8c646dfbca3e4f6806b2441"
server_key_version: "encryptionkey@0"
I20260620 14:48:46.862944 14336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:48:46.863649 14351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:46.863811 14336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:48:46.863889 14336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "7c0d5fdb8bce49d994469114de53dd14"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-cx89"
server_key: "48e2afd9fd9fde61317ff9650154caab"
server_key_iv: "152bf0a8c8c646dfbca3e4f6806b2441"
server_key_version: "encryptionkey@0"
I20260620 14:48:46.863955 14336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:46.880179 14336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:46.882009 14336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:46.882148 14336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:46.882350 14336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:46.882627 14336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:46.882659 14336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:46.882680 14336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:46.882702 14336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:46.887925 14336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41269
I20260620 14:48:46.887986 14464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41269 every 8 connection(s)
I20260620 14:48:46.888269 14336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:48:46 dist-test-slave-cx89 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:48:46.894213 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 14336
I20260620 14:48:46.894302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:46.894434 392 external_mini_cluster.cc:1428] Setting key 62c885f3d7b5f44b1b55d34f2b7ee081
I20260620 14:48:46.896708 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:46.888939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58015 (local address 127.0.98.62:41253)
0620 14:48:46.888987 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:46.888989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:46.889180 (+ 191us) server_negotiation.cc:407] Connection header received
0620 14:48:46.889640 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:46.889641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:46.889650 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:46.889696 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:46.890564 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.891047 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.891998 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.892185 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.892229 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:46.894092 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:46.894094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:46.894094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:46.894101 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:46.895379 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:46.895814 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:46.895814 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:46.895814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:46.895861 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:46.896091 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:46.896092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:46.896092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:46.896184 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:46.896240 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:46.896536 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 14:48:46.896611 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:48:46.896790 14467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:46.888913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:58015)
0620 14:48:46.889068 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:46.889104 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:46.889568 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:46.889751 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:46.889754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:46.890293 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:46.890499 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:46.890504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.891112 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.891113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:46.891934 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:46.891936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.892235 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.892235 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:46.892326 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:46.892735 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:46.892743 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:46.894004 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:46.895471 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:46.895478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:46.895481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:46.895754 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:46.895910 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:46.895911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:46.895911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:46.896030 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:46.896236 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:46.896241 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:46.896407 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 14:48:46.896502 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:48:46.896622 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260620 14:48:46.897218 14465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41253
I20260620 14:48:46.897313 14465 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:46.897487 14465 heartbeater.cc:511] Master 127.0.98.62:41253 requested a full tablet report, sending...
I20260620 14:48:46.897846 14140 ts_manager.cc:194] Registered new tserver with Master: 7c0d5fdb8bce49d994469114de53dd14 (127.0.98.2:41269)
I20260620 14:48:46.898191 14140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58015
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:48:46.916386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41253
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:46.993108 14470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:46.993299 14470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:46.993319 14470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:46.994413 14470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:46.994454 14470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:46.994478 14470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:48:46.995899 14470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:46.996110 14470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:46.996311 14470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:47.000957 14476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:47.000938 14475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:47.001142 14478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:47.001042 14470 server_base.cc:1034] running on GCE node
I20260620 14:48:47.001327 14470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:47.001531 14470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:47.002660 14470 hybrid_clock.cc:648] HybridClock initialized: now 1781966927002650 us; error 29 us; skew 500 ppm
Jun 20 14:48:47 dist-test-slave-cx89 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:47.004971 14470 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:48:47.005378 14470 webserver.cc:460] Webserver started at http://127.0.98.3:44143/ using document root <none> and password file <none>
I20260620 14:48:47.005558 14470 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:47.005595 14470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:47.005688 14470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:47.006619 14470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "40d7a8ee7fbe49a6b990cb1692d6761b"
format_stamp: "Formatted at 2026-06-20 14:48:47 on dist-test-slave-cx89"
server_key: "6c908a8efe103ad1560bf1f101b45766"
server_key_iv: "0163cfa08152e090e95c513ab9607cd2"
server_key_version: "encryptionkey@0"
I20260620 14:48:47.006899 14470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "40d7a8ee7fbe49a6b990cb1692d6761b"
format_stamp: "Formatted at 2026-06-20 14:48:47 on dist-test-slave-cx89"
server_key: "6c908a8efe103ad1560bf1f101b45766"
server_key_iv: "0163cfa08152e090e95c513ab9607cd2"
server_key_version: "encryptionkey@0"
I20260620 14:48:47.008061 14470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:48:47.008769 14485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:47.008952 14470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:47.009029 14470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "40d7a8ee7fbe49a6b990cb1692d6761b"
format_stamp: "Formatted at 2026-06-20 14:48:47 on dist-test-slave-cx89"
server_key: "6c908a8efe103ad1560bf1f101b45766"
server_key_iv: "0163cfa08152e090e95c513ab9607cd2"
server_key_version: "encryptionkey@0"
I20260620 14:48:47.009083 14470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:47.021688 14470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:47.023609 14470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:47.023716 14470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:47.023896 14470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:47.024144 14470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:47.024173 14470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:47.024194 14470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:47.024209 14470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:47.029587 14470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35203
I20260620 14:48:47.029630 14598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35203 every 8 connection(s)
I20260620 14:48:47.029918 14470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:47.030478 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 14470
I20260620 14:48:47.030560 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:47.030751 392 external_mini_cluster.cc:1428] Setting key 46baa0a4d43a10fb7c21dbdb2b9e7d4c
Jun 20 14:48:47 dist-test-slave-cx89 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:48:47.038626 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.030592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44029 (local address 127.0.98.62:41253)
0620 14:48:47.030648 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.030650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.030858 (+ 208us) server_negotiation.cc:407] Connection header received
0620 14:48:47.031543 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.031545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.031558 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.031609 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:47.032502 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.032992 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.033953 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.034154 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.034208 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.036253 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:47.036254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:47.036254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:47.036262 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:47.037421 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:47.037787 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:47.037788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:47.037788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:47.037841 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:47.038043 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:47.038044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:47.038044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:47.038161 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:47.038216 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.038461 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.038536 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:48:47.038806 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.030550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:44029)
0620 14:48:47.030743 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.030785 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:47.031380 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:47.031682 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:47.031687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:47.032231 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:47.032415 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.032420 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.033069 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.033070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.033877 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.033879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.034219 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.034220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.034345 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.034803 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:47.034811 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:47.036163 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:47.037496 (+ 1333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:47.037502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:47.037504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:47.037722 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:47.037894 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:47.037894 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:47.037895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:47.037990 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:47.038214 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:47.038220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:47.038349 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 14:48:47.038436 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:48:47.038596 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
I20260620 14:48:47.039263 14599 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41253
I20260620 14:48:47.039340 14599 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:47.039520 14599 heartbeater.cc:511] Master 127.0.98.62:41253 requested a full tablet report, sending...
I20260620 14:48:47.039880 14140 ts_manager.cc:194] Registered new tserver with Master: 40d7a8ee7fbe49a6b990cb1692d6761b (127.0.98.3:35203)
I20260620 14:48:47.040234 14140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44029
I20260620 14:48:47.042404 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:48:47.049855 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.043818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43334 (local address 127.0.98.62:41253)
0620 14:48:47.043879 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.043880 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.043997 (+ 117us) server_negotiation.cc:407] Connection header received
0620 14:48:47.044067 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.044068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.044079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.044136 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:47.044744 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.045226 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.046018 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.046224 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.046296 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.047617 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:47.047619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:47.047619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:47.047627 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:47.048741 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:47.049067 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:47.049068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:47.049068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:47.049125 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:47.049303 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:47.049303 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:47.049304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:47.049460 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:47.049557 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.049679 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.049761 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:48:47.051293 14140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:48:47.052909 14140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:47.058656 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.054883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39717 (local address 127.0.0.1:41456)
0620 14:48:47.055105 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.055114 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:47.055154 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:47.055761 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:47.055762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:47.055764 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:47.055815 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.055817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.056708 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.056708 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.057463 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.057466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.058363 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.058364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.058449 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.058465 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 14:48:47.058526 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:48:47.058576 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":58,"threads_started":1}
I20260620 14:48:47.058810 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.054957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.0.98.2:41269)
0620 14:48:47.055275 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.055280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.055289 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:48:47.055316 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.055318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.055329 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.055383 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:47.055587 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.056388 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.057935 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.058451 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.058522 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.058577 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.058686 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.058731 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":74,"threads_started":1}
I20260620 14:48:47.059204 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.055137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41456 (local address 127.0.98.1:39717)
0620 14:48:47.055300 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.055304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.055315 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:48:47.055356 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.055358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.055373 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.055679 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:47.055934 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.056591 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.057551 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.058272 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.058355 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.058423 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.059090 (+ 667us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.059132 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":65,"threads_started":1}
I20260620 14:48:47.058789 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.054865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41269 (local address 127.0.0.1:45368)
0620 14:48:47.055050 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.055088 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:47.055153 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:47.055407 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:47.055410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:47.055437 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:47.055516 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.055518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.056481 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.056482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.057851 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.057854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.058533 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.058534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.058602 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.058606 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:48:47.058672 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:48:47.058714 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":74,"threads_started":1}
I20260620 14:48:47.060071 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.054883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35203 (local address 127.0.0.1:52454)
0620 14:48:47.055515 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.055525 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:47.055578 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:47.056651 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:47.056652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:47.056654 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:47.056707 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.056708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.057896 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.057897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.058885 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.058888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.059859 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.059860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.059919 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.059923 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:48:47.059969 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 14:48:47.060005 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":12,"thread_start_us":88,"threads_started":1}
I20260620 14:48:47.060132 14265 tablet_service.cc:1467] Processing CreateTablet for tablet 05b75fd137d64a2d984e4ccf671b19e8 (DEFAULT_TABLE table=test-table [id=b35f164a59d542309bc990bddd7c3f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:47.060215 14399 tablet_service.cc:1467] Processing CreateTablet for tablet 05b75fd137d64a2d984e4ccf671b19e8 (DEFAULT_TABLE table=test-table [id=b35f164a59d542309bc990bddd7c3f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:47.060372 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.055121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.0.98.3:35203)
0620 14:48:47.055300 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.055304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.056497 (+ 1193us) server_negotiation.cc:407] Connection header received
0620 14:48:47.056541 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.056543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.056559 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.056631 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:47.056778 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.057808 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.058993 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.059688 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.059751 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.059801 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.060259 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.060301 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
I20260620 14:48:47.060523 14265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b75fd137d64a2d984e4ccf671b19e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:47.060534 14399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b75fd137d64a2d984e4ccf671b19e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:47.061053 14533 tablet_service.cc:1467] Processing CreateTablet for tablet 05b75fd137d64a2d984e4ccf671b19e8 (DEFAULT_TABLE table=test-table [id=b35f164a59d542309bc990bddd7c3f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:47.061337 14533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b75fd137d64a2d984e4ccf671b19e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:47.062884 14613 tablet_bootstrap.cc:492] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14: Bootstrap starting.
I20260620 14:48:47.063361 14613 tablet_bootstrap.cc:654] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:47.063827 14614 tablet_bootstrap.cc:492] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52: Bootstrap starting.
I20260620 14:48:47.064098 14613 log.cc:824] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:47.064445 14614 tablet_bootstrap.cc:654] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:47.064965 14613 tablet_bootstrap.cc:492] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14: No bootstrap required, opened a new log
I20260620 14:48:47.064956 14614 log.cc:824] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:47.064960 14616 tablet_bootstrap.cc:492] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b: Bootstrap starting.
I20260620 14:48:47.065114 14613 ts_tablet_manager.cc:1397] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:48:47.065560 14616 tablet_bootstrap.cc:654] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:47.065677 14614 tablet_bootstrap.cc:492] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52: No bootstrap required, opened a new log
I20260620 14:48:47.065744 14614 ts_tablet_manager.cc:1397] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:48:47.066033 14616 log.cc:824] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:47.066706 14616 tablet_bootstrap.cc:492] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b: No bootstrap required, opened a new log
I20260620 14:48:47.066768 14616 ts_tablet_manager.cc:1397] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:48:47.067086 14613 raft_consensus.cc:348] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:47.067263 14613 raft_consensus.cc:374] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:47.067297 14613 raft_consensus.cc:729] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0d5fdb8bce49d994469114de53dd14, State: Initialized, Role: FOLLOWER
I20260620 14:48:47.067497 14613 consensus_queue.cc:260] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:47.067615 14614 raft_consensus.cc:348] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:47.067811 14614 raft_consensus.cc:374] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:47.067868 14614 raft_consensus.cc:729] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11b228dfaee14cb484bb91c91a417a52, State: Initialized, Role: FOLLOWER
I20260620 14:48:47.067883 14613 ts_tablet_manager.cc:1428] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:47.067997 14465 heartbeater.cc:503] Master 127.0.98.62:41253 was elected leader, sending a full tablet report...
I20260620 14:48:47.068014 14614 consensus_queue.cc:260] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:47.068238 14614 ts_tablet_manager.cc:1428] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:48:47.068264 14331 heartbeater.cc:503] Master 127.0.98.62:41253 was elected leader, sending a full tablet report...
I20260620 14:48:47.068820 14616 raft_consensus.cc:348] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:47.068980 14616 raft_consensus.cc:374] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:47.069059 14616 raft_consensus.cc:729] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d7a8ee7fbe49a6b990cb1692d6761b, State: Initialized, Role: FOLLOWER
I20260620 14:48:47.069180 14616 consensus_queue.cc:260] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:47.069442 14616 ts_tablet_manager.cc:1428] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:47.069473 14599 heartbeater.cc:503] Master 127.0.98.62:41253 was elected leader, sending a full tablet report...
I20260620 14:48:47.109283 14620 raft_consensus.cc:482] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:47.109397 14620 raft_consensus.cc:504] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:47.109764 14620 leader_election.cc:288] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0d5fdb8bce49d994469114de53dd14 (127.0.98.2:41269), 40d7a8ee7fbe49a6b990cb1692d6761b (127.0.98.3:35203)
I20260620 14:48:47.112730 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.109996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37895 (local address 127.0.98.3:35203)
0620 14:48:47.110042 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.110044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.110053 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:48:47.110081 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.110082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.110095 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.110141 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:47.110298 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.110823 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.111837 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.112399 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.112454 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.112463 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.112622 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.112661 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:48:47.112735 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.109927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35203 (local address 127.0.98.1:37895)
0620 14:48:47.109979 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.109990 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:47.110029 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:47.110148 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:47.110149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:47.110151 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:47.110226 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.110228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.110891 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.110892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.111740 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.111742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.112480 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.112481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.112562 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.112565 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:48:47.112606 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 14:48:47.112637 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:48:47.112761 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.110070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49129 (local address 127.0.98.2:41269)
0620 14:48:47.110203 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.110205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.110212 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:48:47.110234 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.110235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.110246 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.110295 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:47.110425 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.110878 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.112050 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.112472 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.112533 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.112544 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.112668 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.112706 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 14:48:47.112808 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.109933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41269 (local address 127.0.98.1:49129)
0620 14:48:47.110115 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.110125 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:47.110166 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:47.110303 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:47.110303 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:47.110305 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:47.110369 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.110370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.110959 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.110959 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.111971 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.111974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.112543 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.112544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.112601 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.112604 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:48:47.112657 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:48:47.112690 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
I20260620 14:48:47.113180 14419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b75fd137d64a2d984e4ccf671b19e8" candidate_uuid: "11b228dfaee14cb484bb91c91a417a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0d5fdb8bce49d994469114de53dd14" is_pre_election: true
I20260620 14:48:47.113174 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b75fd137d64a2d984e4ccf671b19e8" candidate_uuid: "11b228dfaee14cb484bb91c91a417a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" is_pre_election: true
I20260620 14:48:47.113329 14553 raft_consensus.cc:2454] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11b228dfaee14cb484bb91c91a417a52 in term 0.
I20260620 14:48:47.113329 14419 raft_consensus.cc:2454] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11b228dfaee14cb484bb91c91a417a52 in term 0.
I20260620 14:48:47.113549 14221 leader_election.cc:302] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11b228dfaee14cb484bb91c91a417a52, 40d7a8ee7fbe49a6b990cb1692d6761b; no voters:
I20260620 14:48:47.113690 14620 raft_consensus.cc:2781] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:47.113744 14620 raft_consensus.cc:482] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:47.113766 14620 raft_consensus.cc:3037] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:47.114404 14620 raft_consensus.cc:504] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:47.114521 14620 leader_election.cc:288] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 1 election: Requested vote from peers 7c0d5fdb8bce49d994469114de53dd14 (127.0.98.2:41269), 40d7a8ee7fbe49a6b990cb1692d6761b (127.0.98.3:35203)
I20260620 14:48:47.114663 14419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b75fd137d64a2d984e4ccf671b19e8" candidate_uuid: "11b228dfaee14cb484bb91c91a417a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0d5fdb8bce49d994469114de53dd14"
I20260620 14:48:47.114665 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b75fd137d64a2d984e4ccf671b19e8" candidate_uuid: "11b228dfaee14cb484bb91c91a417a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b"
I20260620 14:48:47.114732 14419 raft_consensus.cc:3037] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:47.114743 14553 raft_consensus.cc:3037] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:47.115388 14419 raft_consensus.cc:2454] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11b228dfaee14cb484bb91c91a417a52 in term 1.
I20260620 14:48:47.115442 14553 raft_consensus.cc:2454] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11b228dfaee14cb484bb91c91a417a52 in term 1.
I20260620 14:48:47.115566 14220 leader_election.cc:302] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11b228dfaee14cb484bb91c91a417a52, 7c0d5fdb8bce49d994469114de53dd14; no voters:
I20260620 14:48:47.115664 14620 raft_consensus.cc:2781] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:47.115795 14620 raft_consensus.cc:686] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [term 1 LEADER]: Becoming Leader. State: Replica: 11b228dfaee14cb484bb91c91a417a52, State: Running, Role: LEADER
I20260620 14:48:47.115880 14620 consensus_queue.cc:237] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:47.116648 14140 catalog_manager.cc:5495] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11b228dfaee14cb484bb91c91a417a52 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "11b228dfaee14cb484bb91c91a417a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } health_report { overall_health: HEALTHY } } }
I20260620 14:48:47.132014 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.129977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41466 (local address 127.0.98.1:39717)
0620 14:48:47.130043 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.130045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.130053 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:48:47.130089 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.130090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.130107 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.130166 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:47.130263 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.130696 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.131377 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.131611 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.131657 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.131771 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:47.131867 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:47.131919 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.131930 (+ 11us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.131946 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:48:47.133705 14419 raft_consensus.cc:1261] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14 [term 1 FOLLOWER]: Refusing update from remote peer 11b228dfaee14cb484bb91c91a417a52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:47.133705 14553 raft_consensus.cc:1261] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b [term 1 FOLLOWER]: Refusing update from remote peer 11b228dfaee14cb484bb91c91a417a52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:47.134063 14623 consensus_queue.cc:1035] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:47.134198 14620 consensus_queue.cc:1035] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:47.136345 14626 mvcc.cc:205] Tried to move back new op lower bound from 7298936533537140736 to 7298936533467164672. Current Snapshot: MvccSnapshot[applied={T|T < 7298936533537140736}]
I20260620 14:48:47.136644 14627 mvcc.cc:205] Tried to move back new op lower bound from 7298936533537140736 to 7298936533467164672. Current Snapshot: MvccSnapshot[applied={T|T < 7298936533537140736}]
I20260620 14:48:47.137028 14628 mvcc.cc:205] Tried to move back new op lower bound from 7298936533537140736 to 7298936533467164672. Current Snapshot: MvccSnapshot[applied={T|T < 7298936533537140736}]
I20260620 14:48:47.138698 14140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:47.138789 14140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:47.139477 14140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07: Sending DeleteTablet for 3 replicas of tablet 05b75fd137d64a2d984e4ccf671b19e8
I20260620 14:48:47.139848 14265 tablet_service.cc:1514] Processing DeleteTablet for tablet 05b75fd137d64a2d984e4ccf671b19e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41456
I20260620 14:48:47.139968 14399 tablet_service.cc:1514] Processing DeleteTablet for tablet 05b75fd137d64a2d984e4ccf671b19e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45368
I20260620 14:48:47.140057 14636 tablet_replica.cc:330] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52: stopping tablet replica
I20260620 14:48:47.140175 14636 raft_consensus.cc:2229] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:47.140208 14637 tablet_replica.cc:330] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14: stopping tablet replica
I20260620 14:48:47.140218 14533 tablet_service.cc:1514] Processing DeleteTablet for tablet 05b75fd137d64a2d984e4ccf671b19e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52454
I20260620 14:48:47.140326 14637 raft_consensus.cc:2229] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:47.140342 14636 raft_consensus.cc:2258] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:47.140408 14637 raft_consensus.cc:2258] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:47.140725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 14202
I20260620 14:48:47.140802 14637 ts_tablet_manager.cc:1905] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:47.140811 14636 ts_tablet_manager.cc:1905] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:47.142436 14637 ts_tablet_manager.cc:1918] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:47.142491 14637 log.cc:1196] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/05b75fd137d64a2d984e4ccf671b19e8
I20260620 14:48:47.142729 14637 ts_tablet_manager.cc:1939] T 05b75fd137d64a2d984e4ccf671b19e8 P 7c0d5fdb8bce49d994469114de53dd14: Deleting consensus metadata
I20260620 14:48:47.143075 14126 catalog_manager.cc:4841] TS 7c0d5fdb8bce49d994469114de53dd14 (127.0.98.2:41269): tablet 05b75fd137d64a2d984e4ccf671b19e8 (table test-table [id=b35f164a59d542309bc990bddd7c3f3b]) successfully deleted
I20260620 14:48:47.143517 14638 tablet_replica.cc:330] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b: stopping tablet replica
I20260620 14:48:47.143622 14638 raft_consensus.cc:2229] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:47.143730 14638 raft_consensus.cc:2258] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:47.144078 14638 ts_tablet_manager.cc:1905] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:47.145277 14638 ts_tablet_manager.cc:1918] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:47.145340 14638 log.cc:1196] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/05b75fd137d64a2d984e4ccf671b19e8
I20260620 14:48:47.145579 14638 ts_tablet_manager.cc:1939] T 05b75fd137d64a2d984e4ccf671b19e8 P 40d7a8ee7fbe49a6b990cb1692d6761b: Deleting consensus metadata
I20260620 14:48:47.145947 14127 catalog_manager.cc:4841] TS 40d7a8ee7fbe49a6b990cb1692d6761b (127.0.98.3:35203): tablet 05b75fd137d64a2d984e4ccf671b19e8 (table test-table [id=b35f164a59d542309bc990bddd7c3f3b]) successfully deleted
W20260620 14:48:47.146246 14125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39717 (error 108)
I20260620 14:48:47.146399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 14336
I20260620 14:48:47.146905 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.146528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39717 (local address 127.0.0.1:41478)
0620 14:48:47.146835 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.146871 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288}
W20260620 14:48:47.147022 14125 catalog_manager.cc:4584] TS 11b228dfaee14cb484bb91c91a417a52 (127.0.98.1:39717): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05b75fd137d64a2d984e4ccf671b19e8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39717: connect: Connection refused (error 111)
I20260620 14:48:47.150573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 14470
I20260620 14:48:47.154386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 14108
I20260620 14:48:47.158603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41253
--webserver_interface=127.0.98.62
--webserver_port=44415
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:47.237318 14640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:47.237510 14640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:47.237530 14640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:47.238600 14640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:47.238632 14640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:47.238644 14640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:47.238655 14640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:47.238665 14640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:47.240090 14640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:47.240276 14640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:47.240433 14640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:47.248198 14649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:47.248167 14647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:47.248208 14640 server_base.cc:1034] running on GCE node
W20260620 14:48:47.248171 14646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:47.248696 14640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:47.248993 14640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:47.250128 14640 hybrid_clock.cc:648] HybridClock initialized: now 1781966927250115 us; error 29 us; skew 500 ppm
Jun 20 14:48:47 dist-test-slave-cx89 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:47.252614 14640 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:48:47.253015 14640 webserver.cc:460] Webserver started at http://127.0.98.62:44415/ using document root <none> and password file <none>
I20260620 14:48:47.253223 14640 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:47.253280 14640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:47.254585 14640 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:47.255153 14656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:47.255311 14640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:48:47.255393 14640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "09c65c0552ec45ccb4901fd5f5283b07"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-cx89"
server_key: "da08776fe610d22b660a847f6ae402fc"
server_key_iv: "b54af5fc5278cee6b60594db91788a57"
server_key_version: "encryptionkey@0"
I20260620 14:48:47.255797 14640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:47.290807 14640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:47.292740 14640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:47.292872 14640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:47.296646 14640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41253
I20260620 14:48:47.296698 14708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41253 every 8 connection(s)
I20260620 14:48:47.297071 14640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:47.298590 14709 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:48:47.299229 14709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07: Bootstrap starting.
I20260620 14:48:47.303399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 14640
I20260620 14:48:47.303743 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39717
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41253
--webserver_port=41889
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:48:47.307312 14709 log.cc:824] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:47.309087 14709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:48:47.309319 14709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07: Bootstrap complete.
I20260620 14:48:47.311362 14709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c65c0552ec45ccb4901fd5f5283b07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260620 14:48:47.311647 14709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c65c0552ec45ccb4901fd5f5283b07, State: Initialized, Role: FOLLOWER
I20260620 14:48:47.311805 14709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c65c0552ec45ccb4901fd5f5283b07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260620 14:48:47.311878 14709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:47.311929 14709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:47.311975 14709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:48:47.312940 14709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c65c0552ec45ccb4901fd5f5283b07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260620 14:48:47.313035 14709 leader_election.cc:302] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09c65c0552ec45ccb4901fd5f5283b07; no voters:
I20260620 14:48:47.313289 14709 leader_election.cc:288] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:48:47.313320 14713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:48:47.313427 14713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 2 LEADER]: Becoming Leader. State: Replica: 09c65c0552ec45ccb4901fd5f5283b07, State: Running, Role: LEADER
I20260620 14:48:47.313510 14713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c65c0552ec45ccb4901fd5f5283b07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260620 14:48:47.313561 14709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:47.313757 14713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09c65c0552ec45ccb4901fd5f5283b07. Latest consensus state: current_term: 2 leader_uuid: "09c65c0552ec45ccb4901fd5f5283b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c65c0552ec45ccb4901fd5f5283b07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } } }
I20260620 14:48:47.313810 14713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:47.313992 14717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:47.314025 14714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "09c65c0552ec45ccb4901fd5f5283b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c65c0552ec45ccb4901fd5f5283b07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } } }
I20260620 14:48:47.314205 14714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:47.314765 14717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:47.314868 14717 catalog_manager.cc:1259] Loaded cluster ID: 9cdefb78af174a258722df1610dd9cf6
I20260620 14:48:47.314904 14717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:47.315156 14717 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 14:48:47.319833 14717 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 14:48:47.322184 14717 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:47.322377 14717 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07: Loaded TSK: 0
I20260620 14:48:47.322608 14717 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:48:47.383661 14711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:47.383852 14711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:47.383883 14711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:47.384999 14711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:47.385040 14711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:47.385075 14711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:48:47.386468 14711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41889
--tserver_master_addrs=127.0.98.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:47.386701 14711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:47.386901 14711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:47.392561 14737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:47.392496 14736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:47.392496 14739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:47.392856 14711 server_base.cc:1034] running on GCE node
I20260620 14:48:47.393041 14711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:47.393263 14711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:47.394430 14711 hybrid_clock.cc:648] HybridClock initialized: now 1781966927394412 us; error 30 us; skew 500 ppm
Jun 20 14:48:47 dist-test-slave-cx89 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:47.396941 14711 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:48:47.397403 14711 webserver.cc:460] Webserver started at http://127.0.98.1:41889/ using document root <none> and password file <none>
I20260620 14:48:47.397607 14711 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:47.397672 14711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:47.399047 14711 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:47.399834 14746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:47.400008 14711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:48:47.400097 14711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "11b228dfaee14cb484bb91c91a417a52"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-cx89"
server_key: "212c10ccd862fc0099bbd347e8e82128"
server_key_iv: "5b639541e85e8882f3c7f63427a82298"
server_key_version: "encryptionkey@0"
I20260620 14:48:47.400425 14711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:47.408864 14711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:47.410769 14711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:47.410892 14711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:47.411085 14711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:47.411518 14753 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:48:47.413354 14711 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:48:47.413452 14711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:48:47.413498 14711 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:48:47.414222 14711 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:48:47.414256 14711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:48:47.414340 14753 tablet_bootstrap.cc:492] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52: Bootstrap starting.
I20260620 14:48:47.416230 14753 log.cc:824] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:47.419047 14753 tablet_bootstrap.cc:492] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:48:47.419330 14753 tablet_bootstrap.cc:492] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52: Bootstrap complete.
I20260620 14:48:47.419512 14753 ts_tablet_manager.cc:1397] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260620 14:48:47.420715 14711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39717
I20260620 14:48:47.420747 14861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39717 every 8 connection(s)
I20260620 14:48:47.421060 14711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:47.421233 14753 raft_consensus.cc:348] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:47.421703 14753 raft_consensus.cc:729] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11b228dfaee14cb484bb91c91a417a52, State: Initialized, Role: FOLLOWER
I20260620 14:48:47.421893 14753 consensus_queue.cc:260] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:47.422163 14753 ts_tablet_manager.cc:1428] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
Jun 20 14:48:47 dist-test-slave-cx89 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:48:47.428712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 14711
I20260620 14:48:47.429075 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41269
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41253
--webserver_port=42193
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:48:47.433476 14864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.421662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:57263)
0620 14:48:47.421867 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.421905 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:47.422048 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:47.422550 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:47.422554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:47.422815 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:47.422996 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.423000 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.424184 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.424186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.424941 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.424943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.425490 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.425491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.425607 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.426123 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:47.426130 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:47.427939 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:47.431588 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:47.431595 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:47.431598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:47.431892 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:47.432188 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:47.432188 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:47.432189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:47.432323 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:47.432785 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:47.432793 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:47.432971 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 14:48:47.433088 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:48:47.433263 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":59,"threads_started":1}
I20260620 14:48:47.434525 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.421714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57263 (local address 127.0.98.62:41253)
0620 14:48:47.422204 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.422212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.422248 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:48:47.422357 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.422362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.422400 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.422512 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:47.423106 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.424084 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.425018 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.425416 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.425510 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.428038 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:47.428042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:47.428042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:47.428052 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:47.431513 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:47.431993 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:47.431996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:47.431996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:47.432125 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:47.432387 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:47.432389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:47.432389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:47.432705 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:47.432786 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.433158 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.434315 (+ 1157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":66,"threads_started":1}
I20260620 14:48:47.435145 14862 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41253
I20260620 14:48:47.435250 14862 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:47.435477 14862 heartbeater.cc:511] Master 127.0.98.62:41253 requested a full tablet report, sending...
I20260620 14:48:47.436275 14673 ts_manager.cc:194] Registered new tserver with Master: 11b228dfaee14cb484bb91c91a417a52 (127.0.98.1:39717)
I20260620 14:48:47.437476 14673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57263
I20260620 14:48:47.438418 14862 heartbeater.cc:503] Master 127.0.98.62:41253 was elected leader, sending a full tablet report...
I20260620 14:48:47.446198 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.437735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41494 (local address 127.0.98.1:39717)
0620 14:48:47.439515 (+ 1780us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.439519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.442434 (+ 2915us) server_negotiation.cc:407] Connection header received
0620 14:48:47.442522 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.442524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.442540 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.442619 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:47.442750 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.443841 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.445391 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.445870 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.445933 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.445980 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.446111 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.446139 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1748,"thread_start_us":87,"threads_started":1}
I20260620 14:48:47.446223 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.436914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39717 (local address 127.0.0.1:41494)
0620 14:48:47.442325 (+ 5411us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.442363 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:47.442446 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:47.442626 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:47.442628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:47.442630 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:47.442693 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.442695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.443924 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.443925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.445300 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.445304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.445944 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.445944 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.446019 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.446033 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 14:48:47.446092 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:48:47.446146 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5378,"thread_start_us":86,"threads_started":1}
I20260620 14:48:47.446617 14780 tablet_service.cc:1514] Processing DeleteTablet for tablet 05b75fd137d64a2d984e4ccf671b19e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41494
I20260620 14:48:47.447273 14870 tablet_replica.cc:330] stopping tablet replica
I20260620 14:48:47.447372 14870 raft_consensus.cc:2229] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:47.447460 14870 raft_consensus.cc:2258] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:47.447736 14870 ts_tablet_manager.cc:1905] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:47.449764 14870 ts_tablet_manager.cc:1918] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:47.449831 14870 log.cc:1196] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/wal/wals/05b75fd137d64a2d984e4ccf671b19e8
I20260620 14:48:47.450039 14870 ts_tablet_manager.cc:1939] T 05b75fd137d64a2d984e4ccf671b19e8 P 11b228dfaee14cb484bb91c91a417a52: Deleting consensus metadata
I20260620 14:48:47.450516 14658 catalog_manager.cc:4841] TS 11b228dfaee14cb484bb91c91a417a52 (127.0.98.1:39717): tablet 05b75fd137d64a2d984e4ccf671b19e8 (table test-table [id=b35f164a59d542309bc990bddd7c3f3b]) successfully deleted
W20260620 14:48:47.510187 14867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:47.510376 14867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:47.510397 14867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:47.511584 14867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:47.511624 14867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:47.511650 14867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:48:47.513070 14867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42193
--tserver_master_addrs=127.0.98.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:47.513265 14867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:47.513463 14867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:47.518263 14876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:47.518299 14875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:47.518306 14878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:47.518318 14867 server_base.cc:1034] running on GCE node
I20260620 14:48:47.518585 14867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:47.518795 14867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:47.519958 14867 hybrid_clock.cc:648] HybridClock initialized: now 1781966927519940 us; error 31 us; skew 500 ppm
Jun 20 14:48:47 dist-test-slave-cx89 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:47.522353 14867 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:48:47.522750 14867 webserver.cc:460] Webserver started at http://127.0.98.2:42193/ using document root <none> and password file <none>
I20260620 14:48:47.522933 14867 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:47.522996 14867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:47.524343 14867 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:47.524892 14885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:47.525056 14867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:48:47.525132 14867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "7c0d5fdb8bce49d994469114de53dd14"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-cx89"
server_key: "48e2afd9fd9fde61317ff9650154caab"
server_key_iv: "152bf0a8c8c646dfbca3e4f6806b2441"
server_key_version: "encryptionkey@0"
I20260620 14:48:47.525439 14867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:47.537556 14867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:47.539593 14867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:47.539732 14867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:47.539923 14867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:47.540197 14867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:47.540228 14867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:47.540257 14867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:47.540280 14867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:47.545699 14867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41269
I20260620 14:48:47.545775 14998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41269 every 8 connection(s)
I20260620 14:48:47.546062 14867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:48:47 dist-test-slave-cx89 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:48:47.554342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 14867
I20260620 14:48:47.554608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35203
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41253
--webserver_port=44143
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:48:47.554664 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.546734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47415 (local address 127.0.98.62:41253)
0620 14:48:47.546788 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.546790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.547007 (+ 217us) server_negotiation.cc:407] Connection header received
0620 14:48:47.547535 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.547536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.547554 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.547608 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:47.548547 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.549044 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.550029 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.550232 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.550295 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.552336 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:47.552337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:47.552338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:47.552346 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:47.553430 (+ 1084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:47.553775 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:47.553776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:47.553776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:47.553824 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:47.554030 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:47.554030 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:47.554031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:47.554139 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:47.554192 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.554480 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.554575 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:48:47.554738 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.546702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:47415)
0620 14:48:47.546871 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.546908 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:47.547433 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:47.547697 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:47.547702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:47.548254 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:47.548443 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.548448 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.549133 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.549134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.549936 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.549938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.550306 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.550308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.550435 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.550883 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:47.550890 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:47.552242 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:47.553501 (+ 1259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:47.553507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:47.553509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:47.553713 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:47.553876 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:47.553877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:47.553877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:47.553977 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:47.554209 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:47.554214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:47.554334 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 14:48:47.554421 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:48:47.554563 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260620 14:48:47.555189 14999 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41253
I20260620 14:48:47.555274 14999 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:47.555464 14999 heartbeater.cc:511] Master 127.0.98.62:41253 requested a full tablet report, sending...
I20260620 14:48:47.555819 14673 ts_manager.cc:194] Registered new tserver with Master: 7c0d5fdb8bce49d994469114de53dd14 (127.0.98.2:41269)
I20260620 14:48:47.556195 14673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47415
W20260620 14:48:47.633529 15002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:47.633718 15002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:47.633749 15002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:47.634846 15002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:47.634883 15002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:47.634917 15002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:48:47.636334 15002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44143
--tserver_master_addrs=127.0.98.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:47.636557 15002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:47.636754 15002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:47.641391 15007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:47.641484 15002 server_base.cc:1034] running on GCE node
W20260620 14:48:47.641398 15008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:47.641391 15010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:47.641897 15002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:47.642102 15002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:47.643226 15002 hybrid_clock.cc:648] HybridClock initialized: now 1781966927643218 us; error 27 us; skew 500 ppm
Jun 20 14:48:47 dist-test-slave-cx89 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:47.645504 15002 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:48:47.645921 15002 webserver.cc:460] Webserver started at http://127.0.98.3:44143/ using document root <none> and password file <none>
I20260620 14:48:47.646107 15002 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:47.646168 15002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:47.647370 15002 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:48:47.647985 15017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:47.648119 15002 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:47.648202 15002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "40d7a8ee7fbe49a6b990cb1692d6761b"
format_stamp: "Formatted at 2026-06-20 14:48:47 on dist-test-slave-cx89"
server_key: "6c908a8efe103ad1560bf1f101b45766"
server_key_iv: "0163cfa08152e090e95c513ab9607cd2"
server_key_version: "encryptionkey@0"
I20260620 14:48:47.648521 15002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:47.658005 15002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:47.659862 15002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:47.659981 15002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:47.660181 15002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:47.660442 15002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:47.660475 15002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:47.660495 15002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:47.660506 15002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:47.666028 15002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35203
I20260620 14:48:47.666077 15130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35203 every 8 connection(s)
I20260620 14:48:47.666379 15002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:47.669225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 15002
Jun 20 14:48:47 dist-test-slave-cx89 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:48:47.675405 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.669523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43346 (local address 127.0.98.62:41253)
0620 14:48:47.669606 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.669608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.669645 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:48:47.669715 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.669716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.669724 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.669773 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:47.670427 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.670898 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.671837 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.672037 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.672099 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.673219 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:47.673220 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:47.673220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:47.673224 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:47.674401 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:47.674671 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:47.674672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:47.674672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:47.674730 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:47.674911 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:47.674912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:47.674912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:47.675036 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:47.675166 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.675227 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.675307 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65,"thread_start_us":44,"threads_started":1}
I20260620 14:48:47.675632 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.667000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36649 (local address 127.0.98.62:41253)
0620 14:48:47.667048 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.667050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.667248 (+ 198us) server_negotiation.cc:407] Connection header received
0620 14:48:47.667770 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.667771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.667781 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.667819 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:47.668672 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.669139 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.670521 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.670709 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.670762 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.672866 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:47.672867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:47.672867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:47.672875 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:47.674335 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:47.674717 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:47.674718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:47.674718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:47.674763 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:47.674993 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:47.674993 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:47.674993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:47.675096 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:47.675156 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.675471 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.675550 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:48:47.675797 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.666994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:36649)
0620 14:48:47.667140 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.667172 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:47.667678 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:47.667888 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:47.667892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:47.668416 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:47.668607 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.668611 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.669234 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.669235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.670448 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.670451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.670780 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.670781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.670905 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.671347 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:47.671357 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:47.672786 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:47.674409 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:47.674415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:47.674417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:47.674644 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:47.674823 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:47.674823 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:47.674823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:47.674933 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:47.675159 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:47.675164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:47.675318 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 14:48:47.675451 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 14:48:47.675598 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260620 14:48:47.676222 15131 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41253
I20260620 14:48:47.676323 15131 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:47.676497 15131 heartbeater.cc:511] Master 127.0.98.62:41253 requested a full tablet report, sending...
I20260620 14:48:47.676812 14673 ts_manager.cc:194] Registered new tserver with Master: 40d7a8ee7fbe49a6b990cb1692d6761b (127.0.98.3:35203)
I20260620 14:48:47.677138 14673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36649
I20260620 14:48:47.687318 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:48:47.689985 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.687688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43352 (local address 127.0.98.62:41253)
0620 14:48:47.687739 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.687741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.687824 (+ 83us) server_negotiation.cc:407] Connection header received
0620 14:48:47.687903 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.687904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.687948 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.688007 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:47.688123 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.688595 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.689314 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.689509 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.689562 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.689706 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:47.689809 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:47.689879 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.689899 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.689918 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:48:47.690233 14673 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:48:47.691740 14673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:47.693542 14780 tablet_service.cc:1467] Processing CreateTablet for tablet cb1e9be70ba9456b91979c2e2691d90d (DEFAULT_TABLE table=test-table [id=384c4633f7584dc4a6b7130ae608c183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:47.693742 14780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb1e9be70ba9456b91979c2e2691d90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:47.695344 14753 tablet_bootstrap.cc:492] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52: Bootstrap starting.
I20260620 14:48:47.695791 14753 tablet_bootstrap.cc:654] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:47.696441 14753 tablet_bootstrap.cc:492] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52: No bootstrap required, opened a new log
I20260620 14:48:47.696494 14753 ts_tablet_manager.cc:1397] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:47.696648 14753 raft_consensus.cc:348] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:47.696697 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.693292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35203 (local address 127.0.0.1:52456)
0620 14:48:47.693336 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.693346 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:47.693381 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:47.693735 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:47.693736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:47.693737 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:47.693778 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.693779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.694811 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.694812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.695626 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.695629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.696480 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.696481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.696536 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.696540 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:48:47.696590 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:48:47.696630 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 14:48:47.696730 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.693369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52456 (local address 127.0.98.3:35203)
0620 14:48:47.693536 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.693541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.693552 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:48:47.693613 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.693615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.693631 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.693843 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:47.693908 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.694733 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.695708 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.696411 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.696468 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.696528 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.696604 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.696641 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260620 14:48:47.696718 14753 raft_consensus.cc:374] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:47.696771 14753 raft_consensus.cc:729] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11b228dfaee14cb484bb91c91a417a52, State: Initialized, Role: FOLLOWER
I20260620 14:48:47.696820 14753 consensus_queue.cc:260] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:47.696910 14753 ts_tablet_manager.cc:1428] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:47.697645 15065 tablet_service.cc:1467] Processing CreateTablet for tablet cb1e9be70ba9456b91979c2e2691d90d (DEFAULT_TABLE table=test-table [id=384c4633f7584dc4a6b7130ae608c183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:47.697826 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.693303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41269 (local address 127.0.0.1:45376)
0620 14:48:47.693701 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.693711 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:47.693748 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:47.693975 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:47.693976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:47.693978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:47.694010 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.694012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.695171 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.695172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.696231 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.696234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.697611 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.697612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.697673 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.697677 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:48:47.697728 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 14:48:47.697766 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":85,"threads_started":1}
I20260620 14:48:47.697865 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.693515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45376 (local address 127.0.98.2:41269)
0620 14:48:47.693820 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.693824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.693835 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:48:47.693873 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.693875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.693891 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.693973 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:47.694077 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.695043 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.696307 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.697539 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.697595 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.697649 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.697740 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.697781 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":96,"threads_started":1}
I20260620 14:48:47.697989 15065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb1e9be70ba9456b91979c2e2691d90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:47.698499 14933 tablet_service.cc:1467] Processing CreateTablet for tablet cb1e9be70ba9456b91979c2e2691d90d (DEFAULT_TABLE table=test-table [id=384c4633f7584dc4a6b7130ae608c183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:47.698730 14933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb1e9be70ba9456b91979c2e2691d90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:47.700730 15141 tablet_bootstrap.cc:492] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b: Bootstrap starting.
I20260620 14:48:47.701347 15141 tablet_bootstrap.cc:654] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:47.701429 15142 tablet_bootstrap.cc:492] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14: Bootstrap starting.
I20260620 14:48:47.701874 15141 log.cc:824] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:47.701963 15142 tablet_bootstrap.cc:654] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:47.702536 15142 log.cc:824] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:47.702642 15141 tablet_bootstrap.cc:492] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b: No bootstrap required, opened a new log
I20260620 14:48:47.702708 15141 ts_tablet_manager.cc:1397] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:48:47.703182 15142 tablet_bootstrap.cc:492] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14: No bootstrap required, opened a new log
I20260620 14:48:47.703233 15142 ts_tablet_manager.cc:1397] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:48:47.704252 15141 raft_consensus.cc:348] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:47.704383 15141 raft_consensus.cc:374] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:47.704413 15141 raft_consensus.cc:729] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d7a8ee7fbe49a6b990cb1692d6761b, State: Initialized, Role: FOLLOWER
I20260620 14:48:47.704523 15141 consensus_queue.cc:260] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:47.704771 15131 heartbeater.cc:503] Master 127.0.98.62:41253 was elected leader, sending a full tablet report...
I20260620 14:48:47.704771 15141 ts_tablet_manager.cc:1428] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:48:47.704758 15142 raft_consensus.cc:348] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:47.704874 15142 raft_consensus.cc:374] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:47.704895 15142 raft_consensus.cc:729] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0d5fdb8bce49d994469114de53dd14, State: Initialized, Role: FOLLOWER
I20260620 14:48:47.704968 15142 consensus_queue.cc:260] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:47.705209 15142 ts_tablet_manager.cc:1428] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:48:47.705240 14999 heartbeater.cc:503] Master 127.0.98.62:41253 was elected leader, sending a full tablet report...
I20260620 14:48:47.790097 15145 raft_consensus.cc:482] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:47.790223 15145 raft_consensus.cc:504] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:47.790628 15145 leader_election.cc:288] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0d5fdb8bce49d994469114de53dd14 (127.0.98.2:41269), 11b228dfaee14cb484bb91c91a417a52 (127.0.98.1:39717)
I20260620 14:48:47.793315 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.790920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54061 (local address 127.0.98.1:39717)
0620 14:48:47.790975 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.790977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.791023 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:48:47.791078 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.791079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.791090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.791141 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:47.791280 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.791735 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.792654 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.793024 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.793085 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.793095 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.793210 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.793247 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:48:47.793316 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.790862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42563 (local address 127.0.98.2:41269)
0620 14:48:47.790906 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.790908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.790917 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:48:47.790947 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.790947 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.790960 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.791035 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:47.791184 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.791717 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.792643 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.793008 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.793059 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.793070 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.793216 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.793252 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:48:47.793331 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.790777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41269 (local address 127.0.98.3:42563)
0620 14:48:47.790832 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.790846 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:47.790891 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:47.791045 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:47.791046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:47.791048 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:47.791119 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.791121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.791790 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.791791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.792557 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.792560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.793077 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.793078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.793153 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.793155 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:48:47.793200 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 14:48:47.793237 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":28}
I20260620 14:48:47.793342 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.790800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39717 (local address 127.0.98.3:54061)
0620 14:48:47.790973 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.790981 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:47.791032 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:47.791155 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:47.791155 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:47.791157 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:47.791226 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.791227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.791798 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.791799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.792568 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.792570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.793095 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.793096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.793148 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.793151 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:48:47.793195 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 14:48:47.793234 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":4,"thread_start_us":101,"threads_started":1}
I20260620 14:48:47.793661 14953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb1e9be70ba9456b91979c2e2691d90d" candidate_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0d5fdb8bce49d994469114de53dd14" is_pre_election: true
I20260620 14:48:47.793838 14953 raft_consensus.cc:2454] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40d7a8ee7fbe49a6b990cb1692d6761b in term 0.
I20260620 14:48:47.793876 14816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb1e9be70ba9456b91979c2e2691d90d" candidate_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11b228dfaee14cb484bb91c91a417a52" is_pre_election: true
I20260620 14:48:47.794034 14816 raft_consensus.cc:2454] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40d7a8ee7fbe49a6b990cb1692d6761b in term 0.
I20260620 14:48:47.794091 15020 leader_election.cc:302] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d7a8ee7fbe49a6b990cb1692d6761b, 7c0d5fdb8bce49d994469114de53dd14; no voters:
I20260620 14:48:47.794225 15145 raft_consensus.cc:2781] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:47.794286 15145 raft_consensus.cc:482] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:47.794302 15145 raft_consensus.cc:3037] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:47.794963 15145 raft_consensus.cc:504] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:47.795077 15145 leader_election.cc:288] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [CANDIDATE]: Term 1 election: Requested vote from peers 7c0d5fdb8bce49d994469114de53dd14 (127.0.98.2:41269), 11b228dfaee14cb484bb91c91a417a52 (127.0.98.1:39717)
I20260620 14:48:47.795218 14816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb1e9be70ba9456b91979c2e2691d90d" candidate_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11b228dfaee14cb484bb91c91a417a52"
I20260620 14:48:47.795238 14953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb1e9be70ba9456b91979c2e2691d90d" candidate_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0d5fdb8bce49d994469114de53dd14"
I20260620 14:48:47.795291 14816 raft_consensus.cc:3037] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:47.795297 14953 raft_consensus.cc:3037] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:47.795979 14953 raft_consensus.cc:2454] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40d7a8ee7fbe49a6b990cb1692d6761b in term 1.
I20260620 14:48:47.796124 15020 leader_election.cc:302] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d7a8ee7fbe49a6b990cb1692d6761b, 7c0d5fdb8bce49d994469114de53dd14; no voters:
I20260620 14:48:47.796231 15145 raft_consensus.cc:2781] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:47.796343 14816 raft_consensus.cc:2454] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40d7a8ee7fbe49a6b990cb1692d6761b in term 1.
I20260620 14:48:47.796379 15145 raft_consensus.cc:686] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [term 1 LEADER]: Becoming Leader. State: Replica: 40d7a8ee7fbe49a6b990cb1692d6761b, State: Running, Role: LEADER
I20260620 14:48:47.796487 15145 consensus_queue.cc:237] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
W20260620 14:48:47.796583 15000 tablet.cc:2369] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:48:47.797281 14672 catalog_manager.cc:5495] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b reported cstate change: term changed from 0 to 1, leader changed from <none> to 40d7a8ee7fbe49a6b990cb1692d6761b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } health_report { overall_health: UNKNOWN } } }
I20260620 14:48:47.814354 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.812048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52468 (local address 127.0.98.3:35203)
0620 14:48:47.812110 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.812112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.812121 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:48:47.812187 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.812188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.812207 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.812268 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:47.812374 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.812841 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.813596 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.813780 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.813829 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.813999 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:47.814138 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:47.814221 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.814238 (+ 17us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.814257 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:48:47.816182 14816 raft_consensus.cc:1261] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Refusing update from remote peer 40d7a8ee7fbe49a6b990cb1692d6761b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:47.816206 14953 raft_consensus.cc:1261] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14 [term 1 FOLLOWER]: Refusing update from remote peer 40d7a8ee7fbe49a6b990cb1692d6761b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:47.816514 15145 consensus_queue.cc:1035] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [LEADER]: Connected to new peer: Peer: permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:47.816654 15148 consensus_queue.cc:1035] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:47.818665 15152 mvcc.cc:205] Tried to move back new op lower bound from 7298936536332398592 to 7298936536255115264. Current Snapshot: MvccSnapshot[applied={T|T < 7298936536332398592}]
I20260620 14:48:47.818994 15153 mvcc.cc:205] Tried to move back new op lower bound from 7298936536332398592 to 7298936536255115264. Current Snapshot: MvccSnapshot[applied={T|T < 7298936536332398592}]
I20260620 14:48:47.820854 14672 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:47.820979 14672 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:47.821688 14672 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07: Sending DeleteTablet for 3 replicas of tablet cb1e9be70ba9456b91979c2e2691d90d
I20260620 14:48:47.821974 15065 tablet_service.cc:1514] Processing DeleteTablet for tablet cb1e9be70ba9456b91979c2e2691d90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52456
I20260620 14:48:47.822047 14780 tablet_service.cc:1514] Processing DeleteTablet for tablet cb1e9be70ba9456b91979c2e2691d90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41494
I20260620 14:48:47.822120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 14711
I20260620 14:48:47.822175 14933 tablet_service.cc:1514] Processing DeleteTablet for tablet cb1e9be70ba9456b91979c2e2691d90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45376
I20260620 14:48:47.822686 15161 tablet_replica.cc:330] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14: stopping tablet replica
I20260620 14:48:47.822773 15161 raft_consensus.cc:2229] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:47.822865 15161 raft_consensus.cc:2258] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:47.823292 15161 ts_tablet_manager.cc:1905] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:47.824962 15161 ts_tablet_manager.cc:1918] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:47.825037 15161 log.cc:1196] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/cb1e9be70ba9456b91979c2e2691d90d
I20260620 14:48:47.825287 15161 ts_tablet_manager.cc:1939] T cb1e9be70ba9456b91979c2e2691d90d P 7c0d5fdb8bce49d994469114de53dd14: Deleting consensus metadata
I20260620 14:48:47.825721 14659 catalog_manager.cc:4841] TS 7c0d5fdb8bce49d994469114de53dd14 (127.0.98.2:41269): tablet cb1e9be70ba9456b91979c2e2691d90d (table test-table [id=384c4633f7584dc4a6b7130ae608c183]) successfully deleted
W20260620 14:48:47.826978 14658 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39717 (error 108)
I20260620 14:48:47.827157 15160 tablet_replica.cc:330] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b: stopping tablet replica
I20260620 14:48:47.827337 15160 raft_consensus.cc:2229] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:47.827409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 14867
I20260620 14:48:47.827468 15160 raft_consensus.cc:2258] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:47.828298 15160 ts_tablet_manager.cc:1905] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:47.829545 15160 ts_tablet_manager.cc:1918] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:47.829605 15160 log.cc:1196] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/cb1e9be70ba9456b91979c2e2691d90d
I20260620 14:48:47.829838 15160 ts_tablet_manager.cc:1939] T cb1e9be70ba9456b91979c2e2691d90d P 40d7a8ee7fbe49a6b990cb1692d6761b: Deleting consensus metadata
I20260620 14:48:47.830183 14660 catalog_manager.cc:4841] TS 40d7a8ee7fbe49a6b990cb1692d6761b (127.0.98.3:35203): tablet cb1e9be70ba9456b91979c2e2691d90d (table test-table [id=384c4633f7584dc4a6b7130ae608c183]) successfully deleted
I20260620 14:48:47.831707 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.831583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39717 (local address 127.0.0.1:41500)
0620 14:48:47.831633 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.831678 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 14:48:47.831826 14658 catalog_manager.cc:4584] TS 11b228dfaee14cb484bb91c91a417a52 (127.0.98.1:39717): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb1e9be70ba9456b91979c2e2691d90d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39717: connect: Connection refused (error 111)
I20260620 14:48:47.833431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 15002
I20260620 14:48:47.837445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 14640
I20260620 14:48:47.841742 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41253
--webserver_interface=127.0.98.62
--webserver_port=44415
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:47.920579 15163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:47.920781 15163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:47.920801 15163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:47.921923 15163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:47.921960 15163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:47.921973 15163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:47.922008 15163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:47.922029 15163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:47.923480 15163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:47.923697 15163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:47.923893 15163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:47.928409 15169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:47.928401 15171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:47.928396 15168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:47.928581 15163 server_base.cc:1034] running on GCE node
I20260620 14:48:47.928855 15163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:47.929095 15163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:47.930223 15163 hybrid_clock.cc:648] HybridClock initialized: now 1781966927930209 us; error 27 us; skew 500 ppm
Jun 20 14:48:47 dist-test-slave-cx89 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:47.932504 15163 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:48:47.932924 15163 webserver.cc:460] Webserver started at http://127.0.98.62:44415/ using document root <none> and password file <none>
I20260620 14:48:47.933112 15163 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:47.933158 15163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:47.934455 15163 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:48:47.935081 15178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:47.935231 15163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:48:47.935319 15163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "09c65c0552ec45ccb4901fd5f5283b07"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-cx89"
server_key: "da08776fe610d22b660a847f6ae402fc"
server_key_iv: "b54af5fc5278cee6b60594db91788a57"
server_key_version: "encryptionkey@0"
I20260620 14:48:47.935698 15163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:47.951588 15163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:47.953658 15163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:47.953790 15163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:47.957506 15163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41253
I20260620 14:48:47.957544 15230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41253 every 8 connection(s)
I20260620 14:48:47.957881 15163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:47.959349 15231 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:48:47.960062 15231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07: Bootstrap starting.
I20260620 14:48:47.966531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 15163
I20260620 14:48:47.966871 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39717
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41253
--webserver_port=41889
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:48:47.968350 15231 log.cc:824] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:47.970724 15231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:48:47.971000 15231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07: Bootstrap complete.
I20260620 14:48:47.973102 15231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c65c0552ec45ccb4901fd5f5283b07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260620 14:48:47.973333 15231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c65c0552ec45ccb4901fd5f5283b07, State: Initialized, Role: FOLLOWER
I20260620 14:48:47.973486 15231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c65c0552ec45ccb4901fd5f5283b07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260620 14:48:47.973549 15231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:47.973578 15231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:47.973623 15231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:48:47.974606 15231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c65c0552ec45ccb4901fd5f5283b07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260620 14:48:47.974694 15231 leader_election.cc:302] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09c65c0552ec45ccb4901fd5f5283b07; no voters:
I20260620 14:48:47.974848 15231 leader_election.cc:288] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:48:47.974900 15235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:48:47.975020 15235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [term 3 LEADER]: Becoming Leader. State: Replica: 09c65c0552ec45ccb4901fd5f5283b07, State: Running, Role: LEADER
I20260620 14:48:47.975113 15231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:47.975106 15235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c65c0552ec45ccb4901fd5f5283b07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260620 14:48:47.975485 15236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "09c65c0552ec45ccb4901fd5f5283b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c65c0552ec45ccb4901fd5f5283b07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } } }
I20260620 14:48:47.975512 15237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09c65c0552ec45ccb4901fd5f5283b07. Latest consensus state: current_term: 3 leader_uuid: "09c65c0552ec45ccb4901fd5f5283b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c65c0552ec45ccb4901fd5f5283b07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } } }
I20260620 14:48:47.975582 15236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:47.975587 15237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09c65c0552ec45ccb4901fd5f5283b07 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:47.975873 15242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:47.976724 15242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:47.976825 15242 catalog_manager.cc:1259] Loaded cluster ID: 9cdefb78af174a258722df1610dd9cf6
I20260620 14:48:47.976857 15242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 14:48:47.977072 15242 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6fbbd4c9ad google::LogMessage::Fail() at ??:0
@ 0x7f6fbbd4f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f6fbbd4c4de google::LogMessage::Flush() at ??:0
@ 0x7f6fbbd4da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6fc3c6d0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f6fc3c768b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6fbdc1a0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6fbdc0f954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6fbea736db start_thread at ??:0
@ 0x7f6fba84a61f clone at ??:0
W20260620 14:48:48.056896 15233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:48.057101 15233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:48.057123 15233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:48.058251 15233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:48.058293 15233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:48.058319 15233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:48:48.059787 15233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41889
--tserver_master_addrs=127.0.98.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:48.060027 15233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:48.060245 15233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:48.065320 15267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:48.065331 15268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:48.065349 15233 server_base.cc:1034] running on GCE node
W20260620 14:48:48.065554 15270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:48.065929 15233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:48.066165 15233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:48.067297 15233 hybrid_clock.cc:648] HybridClock initialized: now 1781966928067285 us; error 30 us; skew 500 ppm
Jun 20 14:48:48 dist-test-slave-cx89 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:48.069826 15233 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:48:48.070236 15233 webserver.cc:460] Webserver started at http://127.0.98.1:41889/ using document root <none> and password file <none>
I20260620 14:48:48.070442 15233 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:48.070489 15233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:48.071790 15233 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:48.072343 15277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:48.072504 15233 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:48:48.072582 15233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "11b228dfaee14cb484bb91c91a417a52"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-cx89"
server_key: "212c10ccd862fc0099bbd347e8e82128"
server_key_iv: "5b639541e85e8882f3c7f63427a82298"
server_key_version: "encryptionkey@0"
I20260620 14:48:48.072923 15233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:48.085582 15233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:48.087571 15233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:48.087695 15233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:48.087883 15233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:48.088284 15284 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:48:48.090147 15233 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:48:48.090201 15233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:48:48.090236 15233 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:48:48.090857 15233 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:48:48.090888 15233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:48:48.090981 15284 tablet_bootstrap.cc:492] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52: Bootstrap starting.
I20260620 14:48:48.097548 15233 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39717
I20260620 14:48:48.097710 15391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39717 every 8 connection(s)
I20260620 14:48:48.097954 15233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:48.101583 15284 log.cc:824] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:48.102210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 15233
I20260620 14:48:48.102495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41269
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41253
--webserver_port=42193
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:48:48.103847 15284 tablet_bootstrap.cc:492] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:48:48.104075 15284 tablet_bootstrap.cc:492] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52: Bootstrap complete.
I20260620 14:48:48.104787 15284 ts_tablet_manager.cc:1397] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.006s
I20260620 14:48:48.105775 15284 raft_consensus.cc:348] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:48.105999 15284 raft_consensus.cc:729] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11b228dfaee14cb484bb91c91a417a52, State: Initialized, Role: FOLLOWER
I20260620 14:48:48.106139 15284 consensus_queue.cc:260] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:48.106374 15284 ts_tablet_manager.cc:1428] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260620 14:48:48.199149 15394 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:48.098633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:53489)
0620 14:48:48.098797 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.098841 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:48.099297 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:48.198959 (+ 99662us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
W20260620 14:48:48.199385 15392 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41253 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41253: Client connection negotiation failed: client connection to 127.0.98.62:41253: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:48:48.199752 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.199632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:57727)
0620 14:48:48.199680 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.199732 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 14:48:48.200093 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.199981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:44059)
0620 14:48:48.200033 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.200065 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 14:48:48.200229 15392 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:48:48.211544 15396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:48.211738 15396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:48.211761 15396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:48.212894 15396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:48.212935 15396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:48.212963 15396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:48:48.214393 15396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42193
--tserver_master_addrs=127.0.98.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:48.214628 15396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:48.214864 15396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:48:48.219617 15396 server_base.cc:1034] running on GCE node
W20260620 14:48:48.219609 15403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:48.219601 15405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:48.219597 15402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:48.219988 15396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:48.220214 15396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:48.221351 15396 hybrid_clock.cc:648] HybridClock initialized: now 1781966928221338 us; error 28 us; skew 500 ppm
Jun 20 14:48:48 dist-test-slave-cx89 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:48.223662 15396 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:48:48.224083 15396 webserver.cc:460] Webserver started at http://127.0.98.2:42193/ using document root <none> and password file <none>
I20260620 14:48:48.224265 15396 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:48.224303 15396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:48.225468 15396 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:48.225939 15412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:48.226097 15396 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:48.226171 15396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "7c0d5fdb8bce49d994469114de53dd14"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-cx89"
server_key: "48e2afd9fd9fde61317ff9650154caab"
server_key_iv: "152bf0a8c8c646dfbca3e4f6806b2441"
server_key_version: "encryptionkey@0"
I20260620 14:48:48.226488 15396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:48.248746 15396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:48.250844 15396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:48.250959 15396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:48.251185 15396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:48.251518 15396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:48.251555 15396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:48.251587 15396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:48.251611 15396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:48.257277 15396 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41269
I20260620 14:48:48.257332 15525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41269 every 8 connection(s)
I20260620 14:48:48.257653 15396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:48.258502 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 15396
I20260620 14:48:48.258744 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.258285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:50745)
0620 14:48:48.258470 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.258552 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260620 14:48:48.258751 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35203
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41253
--webserver_port=44143
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:48.259150 15526 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41253 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41253: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
I20260620 14:48:48.259400 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.259315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:40379)
0620 14:48:48.259356 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.259381 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 14:48:48.259714 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.259629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:51449)
0620 14:48:48.259664 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.259685 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 14:48:48.259837 15526 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:48:48.337236 15529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:48.337430 15529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:48.337450 15529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:48.338523 15529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:48.338554 15529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:48.338578 15529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:48:48.339965 15529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44143
--tserver_master_addrs=127.0.98.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:48:48.340145 15529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:48.340318 15529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:48:48.345144 15529 server_base.cc:1034] running on GCE node
W20260620 14:48:48.345090 15534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:48.345121 15537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:48.345090 15535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:48.345454 15529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:48.345700 15529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:48.346854 15529 hybrid_clock.cc:648] HybridClock initialized: now 1781966928346842 us; error 29 us; skew 500 ppm
Jun 20 14:48:48 dist-test-slave-cx89 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:48.349258 15529 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:48:48.349668 15529 webserver.cc:460] Webserver started at http://127.0.98.3:44143/ using document root <none> and password file <none>
I20260620 14:48:48.349876 15529 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:48.349924 15529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:48.351218 15529 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:48:48.351842 15544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:48.352008 15529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:48.352088 15529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "40d7a8ee7fbe49a6b990cb1692d6761b"
format_stamp: "Formatted at 2026-06-20 14:48:47 on dist-test-slave-cx89"
server_key: "6c908a8efe103ad1560bf1f101b45766"
server_key_iv: "0163cfa08152e090e95c513ab9607cd2"
server_key_version: "encryptionkey@0"
I20260620 14:48:48.352413 15529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:48.364974 15529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:48.366758 15529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:48.366885 15529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:48.367086 15529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:48.367355 15529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:48.367385 15529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:48.367431 15529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:48.367453 15529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:48.372771 15529 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35203
I20260620 14:48:48.372828 15657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35203 every 8 connection(s)
I20260620 14:48:48.373159 15529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:48.374152 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.373761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:60953)
0620 14:48:48.373933 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.374008 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
W20260620 14:48:48.374553 15658 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41253 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41253: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
I20260620 14:48:48.374802 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.374717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:43609)
0620 14:48:48.374760 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.374784 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 14:48:48.375032 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.374958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:54673)
0620 14:48:48.374993 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.375014 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 14:48:48.375118 15658 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:48:48.383373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 15529
I20260620 14:48:49.200867 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:49.200633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:43597)
0620 14:48:49.200761 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:49.200803 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260620 14:48:49.260325 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:49.260130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:39257)
0620 14:48:49.260237 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:49.260274 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20260620 14:48:49.375613 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:49.375396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:58753)
0620 14:48:49.375524 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:49.375559 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260620 14:48:49.605063 15665 raft_consensus.cc:482] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:49.605197 15665 raft_consensus.cc:504] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:49.605489 15665 leader_election.cc:288] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40d7a8ee7fbe49a6b990cb1692d6761b (127.0.98.3:35203), 7c0d5fdb8bce49d994469114de53dd14 (127.0.98.2:41269)
Jun 20 14:48:49 dist-test-slave-cx89 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 20 14:48:49 dist-test-slave-cx89 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 14:48:49.619344 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:49.605654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41269 (local address 127.0.98.1:41489)
0620 14:48:49.605776 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:49.605784 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:49.605834 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:49.606569 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:49.606572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:49.607044 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:49.607253 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:49.607260 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.608307 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.608308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:49.609098 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:49.609101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.609642 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.609643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:49.609748 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:49.614175 (+ 4427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:49.614184 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:49.616068 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:49.617836 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:49.617842 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:49.617845 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:49.618168 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:49.618386 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:49.618387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:49.618387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:49.618491 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:49.618842 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:49.618848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:49.618974 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 14:48:49.619059 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:48:49.619245 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":46,"thread_start_us":60,"threads_started":1}
I20260620 14:48:49.619360 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:49.605789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41489 (local address 127.0.98.2:41269)
0620 14:48:49.605940 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:49.605948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:49.605971 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:48:49.606377 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:49.606379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:49.606413 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:49.606897 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:49.607356 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.608218 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.609175 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.609562 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.609661 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:49.616165 (+ 6504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:49.616169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:49.616169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:49.616178 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:49.617760 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:49.618242 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:49.618245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:49.618246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:49.618330 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:49.618545 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:49.618546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:49.618547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:49.618780 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:49.618851 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:49.619117 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 14:48:49.619240 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
I20260620 14:48:49.619364 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:49.605744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39521 (local address 127.0.98.3:35203)
0620 14:48:49.605874 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:49.605887 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:49.605918 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:48:49.606346 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:49.606349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:49.606390 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:49.606496 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:49.607374 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.608240 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.609180 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.609562 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.609658 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:49.615895 (+ 6237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:49.615900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:49.615900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:49.615915 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:49.617767 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:49.618251 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:49.618254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:49.618254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:49.618344 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:49.618558 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:49.618559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:49.618559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:49.618787 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:49.618863 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:49.619130 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 14:48:49.619250 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":40,"threads_started":1}
I20260620 14:48:49.620299 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:49.605622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35203 (local address 127.0.98.1:39521)
0620 14:48:49.605686 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:49.605698 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:49.605754 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:49.606569 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:49.606572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:49.607045 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:49.607253 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:49.607260 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.608326 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.608327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:49.609099 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:49.609101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.609639 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.609640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:49.609746 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:49.614176 (+ 4430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:49.614185 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:49.615760 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:49.617863 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:49.617864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:49.617864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:49.618173 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:49.618407 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:49.618408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:49.618408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:49.618498 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:49.618854 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:49.618855 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:49.618975 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 14:48:49.619060 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:48:49.619320 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":183}
W20260620 14:48:49.620383 15280 leader_election.cc:341] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c0d5fdb8bce49d994469114de53dd14 (127.0.98.2:41269): Not found: Tablet not found: cb1e9be70ba9456b91979c2e2691d90d
W20260620 14:48:49.620905 15281 leader_election.cc:341] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40d7a8ee7fbe49a6b990cb1692d6761b (127.0.98.3:35203): Not found: Tablet not found: cb1e9be70ba9456b91979c2e2691d90d
I20260620 14:48:49.620982 15281 leader_election.cc:302] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11b228dfaee14cb484bb91c91a417a52; no voters: 40d7a8ee7fbe49a6b990cb1692d6761b, 7c0d5fdb8bce49d994469114de53dd14
I20260620 14:48:49.621095 15665 raft_consensus.cc:2726] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:48:50.201515 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:50.201288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:35109)
0620 14:48:50.201434 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:50.201471 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":64,"threads_started":1}
I20260620 14:48:50.260933 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:50.260737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:40405)
0620 14:48:50.260856 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:50.260897 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":53,"threads_started":1}
I20260620 14:48:50.376296 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:50.376078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:35721)
0620 14:48:50.376215 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:50.376255 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260620 14:48:51.202205 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:51.201975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:55153)
0620 14:48:51.202115 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:51.202156 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260620 14:48:51.261511 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:51.261303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:34287)
0620 14:48:51.261436 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:51.261471 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
I20260620 14:48:51.376915 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:51.376650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:52509)
0620 14:48:51.376836 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:51.376873 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":81,"threads_started":1}
I20260620 14:48:51.641979 15675 raft_consensus.cc:482] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:51.642087 15675 raft_consensus.cc:504] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:51.642297 15675 leader_election.cc:288] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40d7a8ee7fbe49a6b990cb1692d6761b (127.0.98.3:35203), 7c0d5fdb8bce49d994469114de53dd14 (127.0.98.2:41269)
W20260620 14:48:51.642670 15281 leader_election.cc:341] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40d7a8ee7fbe49a6b990cb1692d6761b (127.0.98.3:35203): Not found: Tablet not found: cb1e9be70ba9456b91979c2e2691d90d
W20260620 14:48:51.642740 15280 leader_election.cc:341] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c0d5fdb8bce49d994469114de53dd14 (127.0.98.2:41269): Not found: Tablet not found: cb1e9be70ba9456b91979c2e2691d90d
I20260620 14:48:51.642817 15280 leader_election.cc:302] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11b228dfaee14cb484bb91c91a417a52; no voters: 40d7a8ee7fbe49a6b990cb1692d6761b, 7c0d5fdb8bce49d994469114de53dd14
I20260620 14:48:51.642899 15675 raft_consensus.cc:2726] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:48:52.202957 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:52.202721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:53349)
0620 14:48:52.202865 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:52.202910 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260620 14:48:52.262143 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:52.261952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:36605)
0620 14:48:52.262065 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:52.262100 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260620 14:48:52.377635 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:52.377423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:35939)
0620 14:48:52.377548 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:52.377582 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":56,"threads_started":1}
I20260620 14:48:53.203708 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:53.203495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:38495)
0620 14:48:53.203613 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:53.203659 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260620 14:48:53.262818 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:53.262595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:33339)
0620 14:48:53.262721 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:53.262774 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260620 14:48:53.378337 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:53.378100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:49015)
0620 14:48:53.378249 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:53.378293 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
I20260620 14:48:54.204383 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:54.204146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:41123)
0620 14:48:54.204305 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:54.204337 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260620 14:48:54.263507 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:54.263269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:41045)
0620 14:48:54.263396 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:54.263456 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260620 14:48:54.379042 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:54.378818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:37475)
0620 14:48:54.378957 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:54.378997 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260620 14:48:54.477339 15685 raft_consensus.cc:482] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:54.477443 15685 raft_consensus.cc:504] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:54.477612 15685 leader_election.cc:288] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40d7a8ee7fbe49a6b990cb1692d6761b (127.0.98.3:35203), 7c0d5fdb8bce49d994469114de53dd14 (127.0.98.2:41269)
W20260620 14:48:54.477954 15281 leader_election.cc:341] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40d7a8ee7fbe49a6b990cb1692d6761b (127.0.98.3:35203): Not found: Tablet not found: cb1e9be70ba9456b91979c2e2691d90d
W20260620 14:48:54.478046 15280 leader_election.cc:341] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c0d5fdb8bce49d994469114de53dd14 (127.0.98.2:41269): Not found: Tablet not found: cb1e9be70ba9456b91979c2e2691d90d
I20260620 14:48:54.478114 15280 leader_election.cc:302] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11b228dfaee14cb484bb91c91a417a52; no voters: 40d7a8ee7fbe49a6b990cb1692d6761b, 7c0d5fdb8bce49d994469114de53dd14
I20260620 14:48:54.478210 15685 raft_consensus.cc:2726] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:48:55.205116 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:55.204892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:60413)
0620 14:48:55.205031 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:55.205070 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
I20260620 14:48:55.264117 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:55.263887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:56947)
0620 14:48:55.264031 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:55.264073 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":60,"threads_started":1}
I20260620 14:48:55.379762 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:55.379552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:36647)
0620 14:48:55.379673 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:55.379714 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260620 14:48:56.205796 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:56.205581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:38465)
0620 14:48:56.205702 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:56.205747 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260620 14:48:56.264748 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:56.264534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:54509)
0620 14:48:56.264674 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:56.264708 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260620 14:48:56.380419 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:56.380195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:38737)
0620 14:48:56.380331 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:56.380373 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
I20260620 14:48:56.490132 15692 raft_consensus.cc:482] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:56.490228 15692 raft_consensus.cc:504] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:48:56.490382 15692 leader_election.cc:288] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40d7a8ee7fbe49a6b990cb1692d6761b (127.0.98.3:35203), 7c0d5fdb8bce49d994469114de53dd14 (127.0.98.2:41269)
W20260620 14:48:56.490741 15281 leader_election.cc:341] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40d7a8ee7fbe49a6b990cb1692d6761b (127.0.98.3:35203): Not found: Tablet not found: cb1e9be70ba9456b91979c2e2691d90d
W20260620 14:48:56.490815 15280 leader_election.cc:341] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c0d5fdb8bce49d994469114de53dd14 (127.0.98.2:41269): Not found: Tablet not found: cb1e9be70ba9456b91979c2e2691d90d
I20260620 14:48:56.490854 15280 leader_election.cc:302] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11b228dfaee14cb484bb91c91a417a52; no voters: 40d7a8ee7fbe49a6b990cb1692d6761b, 7c0d5fdb8bce49d994469114de53dd14
I20260620 14:48:56.490912 15692 raft_consensus.cc:2726] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:48:57.206442 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:57.206223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:47983)
0620 14:48:57.206364 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:57.206398 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260620 14:48:57.265414 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:57.265185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:33421)
0620 14:48:57.265334 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:57.265372 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":72,"threads_started":1}
I20260620 14:48:57.381078 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:57.380854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:41737)
0620 14:48:57.380993 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:57.381033 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260620 14:48:58.207183 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:58.206939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:37997)
0620 14:48:58.207089 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:58.207133 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
I20260620 14:48:58.266090 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:58.265837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:33687)
0620 14:48:58.266005 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:58.266045 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":81,"threads_started":1}
I20260620 14:48:58.381728 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:58.381516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:37825)
0620 14:48:58.381646 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:58.381683 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260620 14:48:59.207845 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:59.207639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:45731)
0620 14:48:59.207750 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:59.207787 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
I20260620 14:48:59.266758 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:59.266549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:57825)
0620 14:48:59.266672 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:59.266714 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260620 14:48:59.382382 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:59.382181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:50135)
0620 14:48:59.382295 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:59.382335 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20260620 14:49:00.208484 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:00.208263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:44725)
0620 14:49:00.208397 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:00.208430 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260620 14:49:00.267566 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:00.267286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:37857)
0620 14:49:00.267477 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:00.267516 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260620 14:49:00.383038 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:00.382816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:59805)
0620 14:49:00.382952 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:00.382994 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260620 14:49:01.209108 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:01.208884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:49191)
0620 14:49:01.209024 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:01.209061 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260620 14:49:01.268321 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:01.268054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:41641)
0620 14:49:01.268235 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:01.268276 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260620 14:49:01.383731 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:01.383525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:58743)
0620 14:49:01.383652 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:01.383686 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260620 14:49:01.630455 15708 raft_consensus.cc:482] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:49:01.630541 15708 raft_consensus.cc:504] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d7a8ee7fbe49a6b990cb1692d6761b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35203 } } peers { permanent_uuid: "7c0d5fdb8bce49d994469114de53dd14" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "11b228dfaee14cb484bb91c91a417a52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39717 } }
I20260620 14:49:01.630676 15708 leader_election.cc:288] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40d7a8ee7fbe49a6b990cb1692d6761b (127.0.98.3:35203), 7c0d5fdb8bce49d994469114de53dd14 (127.0.98.2:41269)
W20260620 14:49:01.631033 15280 leader_election.cc:341] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c0d5fdb8bce49d994469114de53dd14 (127.0.98.2:41269): Not found: Tablet not found: cb1e9be70ba9456b91979c2e2691d90d
W20260620 14:49:01.631124 15281 leader_election.cc:341] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40d7a8ee7fbe49a6b990cb1692d6761b (127.0.98.3:35203): Not found: Tablet not found: cb1e9be70ba9456b91979c2e2691d90d
I20260620 14:49:01.631181 15281 leader_election.cc:302] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11b228dfaee14cb484bb91c91a417a52; no voters: 40d7a8ee7fbe49a6b990cb1692d6761b, 7c0d5fdb8bce49d994469114de53dd14
I20260620 14:49:01.631244 15708 raft_consensus.cc:2726] T cb1e9be70ba9456b91979c2e2691d90d P 11b228dfaee14cb484bb91c91a417a52 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:49:02.209729 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:02.209530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:48521)
0620 14:49:02.209644 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:02.209684 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260620 14:49:02.269061 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:02.268809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:41753)
0620 14:49:02.268970 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:02.269015 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260620 14:49:02.384389 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:02.384144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:38495)
0620 14:49:02.384303 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:02.384344 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260620 14:49:03.210400 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:03.210176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:34751)
0620 14:49:03.210318 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:03.210356 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":57,"threads_started":1}
I20260620 14:49:03.269960 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:03.269709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:45259)
0620 14:49:03.269873 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:03.269914 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
I20260620 14:49:03.385152 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:03.384910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:53897)
0620 14:49:03.385066 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:03.385107 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260620 14:49:03.385975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 15233
I20260620 14:49:03.390192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 15396
I20260620 14:49:03.394085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 15529
2026-06-20T14:49:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (17995 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:49:03 dist-test-slave-cx89 krb5kdc[15717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:49:03 dist-test-slave-cx89 krb5kdc[15717](info): set up 2 sockets
Jun 20 14:49:03 dist-test-slave-cx89 krb5kdc[15717](info): commencing operation
krb5kdc: starting...
W20260620 14:49:04.367206 392 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:49:04 dist-test-slave-cx89 krb5kdc[15717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:49:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:49:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:49:04.475693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40171
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:49:04.553013 15730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:04.553227 15730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:04.553256 15730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:04.554371 15730 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:49:04.554406 15730 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:49:04.554421 15730 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:49:04.554436 15730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:49:04.554451 15730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:04.554467 15730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:04.554487 15730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:49:04.554510 15730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:49:04.555964 15730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:49:04.556202 15730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:04.556396 15730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:04.561090 15736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:04.561115 15738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:04.561163 15730 server_base.cc:1034] running on GCE node
W20260620 14:49:04.561131 15735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:04.561451 15730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:04.561724 15730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:04.562873 15730 hybrid_clock.cc:648] HybridClock initialized: now 1781966944562854 us; error 32 us; skew 500 ppm
Jun 20 14:49:04 dist-test-slave-cx89 krb5kdc[15717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:04.565093 15730 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:49:04.565474 15730 webserver.cc:460] Webserver started at http://127.0.98.62:37505/ using document root <none> and password file <none>
I20260620 14:49:04.565685 15730 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:04.565727 15730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:04.565843 15730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:49:04.566821 15730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "080535b714844b028b54559fceca82a0"
format_stamp: "Formatted at 2026-06-20 14:49:04 on dist-test-slave-cx89"
server_key: "45ece8f915c105a9d37fa7d7fa836567"
server_key_iv: "d8d04417ab01d4e6cbb15a0287bd8c51"
server_key_version: "encryptionkey@0"
I20260620 14:49:04.567128 15730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "080535b714844b028b54559fceca82a0"
format_stamp: "Formatted at 2026-06-20 14:49:04 on dist-test-slave-cx89"
server_key: "45ece8f915c105a9d37fa7d7fa836567"
server_key_iv: "d8d04417ab01d4e6cbb15a0287bd8c51"
server_key_version: "encryptionkey@0"
I20260620 14:49:04.568397 15730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:49:04.569061 15745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:04.569271 15730 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:49:04.569356 15730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "080535b714844b028b54559fceca82a0"
format_stamp: "Formatted at 2026-06-20 14:49:04 on dist-test-slave-cx89"
server_key: "45ece8f915c105a9d37fa7d7fa836567"
server_key_iv: "d8d04417ab01d4e6cbb15a0287bd8c51"
server_key_version: "encryptionkey@0"
I20260620 14:49:04.569423 15730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:04.596256 15730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:04.598274 15730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:04.598388 15730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:04.602102 15730 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40171
I20260620 14:49:04.602149 15797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40171 every 8 connection(s)
I20260620 14:49:04.602458 15730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:49:04.603331 15798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:49:04.605510 15798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: Bootstrap starting.
I20260620 14:49:04.606022 15798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: Neither blocks nor log segments found. Creating new log.
I20260620 14:49:04.606256 15798 log.cc:824] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:04.606843 15798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: No bootstrap required, opened a new log
I20260620 14:49:04.608357 15798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080535b714844b028b54559fceca82a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40171 } }
I20260620 14:49:04.608491 15798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:49:04.608520 15798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 080535b714844b028b54559fceca82a0, State: Initialized, Role: FOLLOWER
I20260620 14:49:04.608623 15798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080535b714844b028b54559fceca82a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40171 } }
I20260620 14:49:04.608670 15798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:49:04.608695 15798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:49:04.608724 15798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:49:04.609180 15798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080535b714844b028b54559fceca82a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40171 } }
I20260620 14:49:04.609263 15798 leader_election.cc:302] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 080535b714844b028b54559fceca82a0; no voters:
I20260620 14:49:04.609385 15798 leader_election.cc:288] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:49:04.609426 15801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:49:04.609592 15798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:49:04.609591 15801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 1 LEADER]: Becoming Leader. State: Replica: 080535b714844b028b54559fceca82a0, State: Running, Role: LEADER
I20260620 14:49:04.609880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 15730
I20260620 14:49:04.609963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:49:04.610132 392 external_mini_cluster.cc:1428] Setting key 6fc6c2d33feb2f83f9558dfdd0a94f4d
I20260620 14:49:04.609715 15801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080535b714844b028b54559fceca82a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40171 } }
I20260620 14:49:04.610846 15802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "080535b714844b028b54559fceca82a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080535b714844b028b54559fceca82a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40171 } } }
I20260620 14:49:04.611048 15803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 080535b714844b028b54559fceca82a0. Latest consensus state: current_term: 1 leader_uuid: "080535b714844b028b54559fceca82a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080535b714844b028b54559fceca82a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40171 } } }
I20260620 14:49:04.611092 15802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:49:04.611094 15803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [sys.catalog]: This master's current role is: LEADER
W20260620 14:49:04.611563 15817 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:49:04.611655 15817 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:49:04.611749 15818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:49:04.611934 15818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:49:04.613432 15818 catalog_manager.cc:1347] Generated new cluster ID: 3326b935460342819d44e6a40743ed1f
I20260620 14:49:04.613481 15818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:49:04 dist-test-slave-cx89 krb5kdc[15717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:49:04.622366 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:04.610803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.0.98.62:40171)
0620 14:49:04.610947 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:04.610956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:04.610990 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:49:04.611102 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:04.611110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:04.611177 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:04.611317 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:04.612180 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:04.613319 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:04.614431 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:04.615075 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:04.615186 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:04.617849 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:04.617853 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:04.617853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:04.617864 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:04.620098 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:04.620605 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:04.620609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:04.620609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:04.620740 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:04.620981 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:04.620982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:04.620982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:04.621395 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:04.621945 (+ 550us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:04.622012 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 14:49:04.622140 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":53,"threads_started":1}
I20260620 14:49:04.627737 15818 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:49:04.628233 15818 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:49:04.631124 15818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: Generated new TSK 0
I20260620 14:49:04.631295 15818 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:49:04.654244 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40171
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:49:04.733294 15824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:04.733487 15824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:04.733508 15824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:04.734637 15824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:04.734678 15824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:04.734702 15824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:49:04.736594 15824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:49:04.736845 15824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:04.737067 15824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:04.742121 15830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:04.742120 15832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:04.742194 15829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:04.742403 15824 server_base.cc:1034] running on GCE node
I20260620 14:49:04.742712 15824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:04.742938 15824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:04.744074 15824 hybrid_clock.cc:648] HybridClock initialized: now 1781966944744061 us; error 26 us; skew 500 ppm
Jun 20 14:49:04 dist-test-slave-cx89 krb5kdc[15717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:04.746534 15824 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:49:04.746945 15824 webserver.cc:460] Webserver started at http://127.0.98.1:43173/ using document root <none> and password file <none>
I20260620 14:49:04.747148 15824 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:04.747193 15824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:04.747315 15824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:49:04.748381 15824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "ce64d8a46f8f4dd0910203fc20d0646f"
format_stamp: "Formatted at 2026-06-20 14:49:04 on dist-test-slave-cx89"
server_key: "3d13e8c51f583dab4c4bad25c01dfbfc"
server_key_iv: "0b43c15d5d092d0abecf103a0750949c"
server_key_version: "encryptionkey@0"
I20260620 14:49:04.748723 15824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ce64d8a46f8f4dd0910203fc20d0646f"
format_stamp: "Formatted at 2026-06-20 14:49:04 on dist-test-slave-cx89"
server_key: "3d13e8c51f583dab4c4bad25c01dfbfc"
server_key_iv: "0b43c15d5d092d0abecf103a0750949c"
server_key_version: "encryptionkey@0"
I20260620 14:49:04.749951 15824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:49:04.750603 15839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:04.750779 15824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:49:04.750859 15824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "ce64d8a46f8f4dd0910203fc20d0646f"
format_stamp: "Formatted at 2026-06-20 14:49:04 on dist-test-slave-cx89"
server_key: "3d13e8c51f583dab4c4bad25c01dfbfc"
server_key_iv: "0b43c15d5d092d0abecf103a0750949c"
server_key_version: "encryptionkey@0"
I20260620 14:49:04.750922 15824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:04.778223 15824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:04.780625 15824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:04.780793 15824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:04.781066 15824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:49:04.781411 15824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:49:04.781453 15824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:04.781487 15824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:49:04.781513 15824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:04.787701 15824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43327
I20260620 14:49:04.787740 15952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43327 every 8 connection(s)
I20260620 14:49:04.788036 15824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:49:04.788933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 15824
I20260620 14:49:04.789031 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:49:04.789209 392 external_mini_cluster.cc:1428] Setting key 1739c2ef357217816661870fea37d1d6
Jun 20 14:49:04 dist-test-slave-cx89 krb5kdc[15717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:49:04.797417 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:04.788712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45535 (local address 127.0.98.62:40171)
0620 14:49:04.788769 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:04.788771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:04.788970 (+ 199us) server_negotiation.cc:407] Connection header received
0620 14:49:04.789493 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:04.789494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:04.789504 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:04.789558 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:04.790517 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:04.791188 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:04.792261 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:04.792477 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:04.792531 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:04.794869 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:04.794870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:04.794871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:04.794878 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:04.796197 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:04.796603 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:04.796604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:04.796605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:04.796679 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:04.796885 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:04.796886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:04.796886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:04.796991 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:04.797041 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:04.797250 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 14:49:04.797318 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:49:04.797530 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:04.788696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:45535)
0620 14:49:04.788847 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:04.788888 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:04.789416 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:04.789637 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:04.789641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:04.790229 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:49:04.790427 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:04.790432 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:04.791275 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:04.791277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:04.792181 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:04.792184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:04.792544 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:04.792545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:04.792685 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:04.793247 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:49:04.793255 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:49:04.794782 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:49:04.796269 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:04.796276 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:04.796279 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:04.796486 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:04.796744 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:04.796744 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:04.796745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:04.796836 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:04.797034 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:49:04.797036 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:49:04.797153 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 14:49:04.797230 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:49:04.797347 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260620 14:49:04.797946 15953 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40171
I20260620 14:49:04.798041 15953 heartbeater.cc:464] Registering TS with master...
I20260620 14:49:04.798218 15953 heartbeater.cc:511] Master 127.0.98.62:40171 requested a full tablet report, sending...
I20260620 14:49:04.798705 15762 ts_manager.cc:194] Registered new tserver with Master: ce64d8a46f8f4dd0910203fc20d0646f (127.0.98.1:43327)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:49:04.799305 15762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:49:04.813412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40171
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:49:04.892220 15958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:04.892431 15958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:04.892453 15958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:04.893546 15958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:04.893579 15958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:04.893604 15958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:49:04.894999 15958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:49:04.895188 15958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:04.895373 15958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:04.900148 15963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:04.900177 15964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:04.900141 15966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:04.900357 15958 server_base.cc:1034] running on GCE node
I20260620 14:49:04.900550 15958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:04.900797 15958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:04.901937 15958 hybrid_clock.cc:648] HybridClock initialized: now 1781966944901924 us; error 29 us; skew 500 ppm
Jun 20 14:49:04 dist-test-slave-cx89 krb5kdc[15717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:04.904358 15958 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:49:04.904803 15958 webserver.cc:460] Webserver started at http://127.0.98.2:45453/ using document root <none> and password file <none>
I20260620 14:49:04.905010 15958 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:04.905058 15958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:04.905179 15958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:49:04.906183 15958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "d60b6c9420004fd488ec636183caea7d"
format_stamp: "Formatted at 2026-06-20 14:49:04 on dist-test-slave-cx89"
server_key: "40a102b1de1284eaed21a42280eecc7d"
server_key_iv: "c756c9e3c1bc91da32ce119fcdff8f07"
server_key_version: "encryptionkey@0"
I20260620 14:49:04.906507 15958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d60b6c9420004fd488ec636183caea7d"
format_stamp: "Formatted at 2026-06-20 14:49:04 on dist-test-slave-cx89"
server_key: "40a102b1de1284eaed21a42280eecc7d"
server_key_iv: "c756c9e3c1bc91da32ce119fcdff8f07"
server_key_version: "encryptionkey@0"
I20260620 14:49:04.907768 15958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:49:04.908479 15973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:04.908628 15958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:49:04.908708 15958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "d60b6c9420004fd488ec636183caea7d"
format_stamp: "Formatted at 2026-06-20 14:49:04 on dist-test-slave-cx89"
server_key: "40a102b1de1284eaed21a42280eecc7d"
server_key_iv: "c756c9e3c1bc91da32ce119fcdff8f07"
server_key_version: "encryptionkey@0"
I20260620 14:49:04.908782 15958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:04.924260 15958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:04.926635 15958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:04.926784 15958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:04.927011 15958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:49:04.927320 15958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:49:04.927361 15958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:04.927388 15958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:49:04.927479 15958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:04.933465 15958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44095
I20260620 14:49:04.933514 16086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44095 every 8 connection(s)
I20260620 14:49:04.933815 15958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:49:04.937557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 15958
I20260620 14:49:04.937656 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:49:04.937860 392 external_mini_cluster.cc:1428] Setting key 6a8b289bf438aec0c70b8e08aac4e657
Jun 20 14:49:04 dist-test-slave-cx89 krb5kdc[15717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:49:04.942387 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:04.934452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53113 (local address 127.0.98.62:40171)
0620 14:49:04.934526 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:04.934528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:04.934682 (+ 154us) server_negotiation.cc:407] Connection header received
0620 14:49:04.935144 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:04.935145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:04.935154 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:04.935187 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:04.936065 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:04.936556 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:04.937487 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:04.937761 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:04.937830 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:04.939928 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:04.939929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:04.939930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:04.939936 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:04.941194 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:04.941537 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:04.941538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:04.941539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:04.941590 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:04.941804 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:04.941805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:04.941806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:04.941911 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:04.941965 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:04.942215 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 14:49:04.942299 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:49:04.942545 16089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:04.934426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:53113)
0620 14:49:04.934570 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:04.934605 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:04.935058 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:04.935242 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:04.935245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:04.935797 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:49:04.935998 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:04.936002 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:04.936617 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:04.936617 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:04.937422 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:04.937424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:04.937831 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:04.937832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:04.937955 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:04.938428 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:49:04.938438 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:49:04.939847 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:49:04.941265 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:04.941272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:04.941274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:04.941476 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:04.941642 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:04.941642 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:04.941643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:04.941751 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:04.941963 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:49:04.941969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:49:04.942102 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 14:49:04.942190 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:49:04.942337 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260620 14:49:04.943045 16087 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40171
I20260620 14:49:04.943132 16087 heartbeater.cc:464] Registering TS with master...
I20260620 14:49:04.943307 16087 heartbeater.cc:511] Master 127.0.98.62:40171 requested a full tablet report, sending...
I20260620 14:49:04.943679 15762 ts_manager.cc:194] Registered new tserver with Master: d60b6c9420004fd488ec636183caea7d (127.0.98.2:44095)
I20260620 14:49:04.944172 15762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53113
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:49:04.960389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40171
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:49:05.037904 16092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:05.038097 16092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:05.038118 16092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:05.039192 16092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:05.039225 16092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:05.039250 16092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:49:05.040658 16092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:49:05.040839 16092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:05.041011 16092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:05.045648 16098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:05.045771 16097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:05.045897 16092 server_base.cc:1034] running on GCE node
W20260620 14:49:05.045892 16100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:05.046180 16092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:05.046444 16092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:05.047574 16092 hybrid_clock.cc:648] HybridClock initialized: now 1781966945047562 us; error 27 us; skew 500 ppm
Jun 20 14:49:05 dist-test-slave-cx89 krb5kdc[15717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:05.049927 16092 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:49:05.050357 16092 webserver.cc:460] Webserver started at http://127.0.98.3:42277/ using document root <none> and password file <none>
I20260620 14:49:05.050562 16092 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:05.050603 16092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:05.050729 16092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:49:05.051784 16092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "57698b8e5272422781fd65d6f72b2619"
format_stamp: "Formatted at 2026-06-20 14:49:05 on dist-test-slave-cx89"
server_key: "82269baba8eeed589205155af8d5dfea"
server_key_iv: "58640e68c623ffd2da51e2321986c11a"
server_key_version: "encryptionkey@0"
I20260620 14:49:05.052093 16092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "57698b8e5272422781fd65d6f72b2619"
format_stamp: "Formatted at 2026-06-20 14:49:05 on dist-test-slave-cx89"
server_key: "82269baba8eeed589205155af8d5dfea"
server_key_iv: "58640e68c623ffd2da51e2321986c11a"
server_key_version: "encryptionkey@0"
I20260620 14:49:05.053329 16092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:49:05.054011 16107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:05.054171 16092 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:49:05.054257 16092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "57698b8e5272422781fd65d6f72b2619"
format_stamp: "Formatted at 2026-06-20 14:49:05 on dist-test-slave-cx89"
server_key: "82269baba8eeed589205155af8d5dfea"
server_key_iv: "58640e68c623ffd2da51e2321986c11a"
server_key_version: "encryptionkey@0"
I20260620 14:49:05.054314 16092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:05.062294 16092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:05.064311 16092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:05.064416 16092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:05.064577 16092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:49:05.064813 16092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:49:05.064841 16092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:05.064862 16092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:49:05.064877 16092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:05.070046 16092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42949
I20260620 14:49:05.070111 16220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42949 every 8 connection(s)
I20260620 14:49:05.070363 16092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:49:05.074596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 16092
I20260620 14:49:05.074702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:49:05.074895 392 external_mini_cluster.cc:1428] Setting key a80cb18182c4c772b82f3f70d2fff5c0
Jun 20 14:49:05 dist-test-slave-cx89 krb5kdc[15717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:49:05.078572 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:05.070982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40907 (local address 127.0.98.62:40171)
0620 14:49:05.071025 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:05.071027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:05.071205 (+ 178us) server_negotiation.cc:407] Connection header received
0620 14:49:05.071789 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:05.071790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:05.071803 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:05.071855 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:05.072764 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.073271 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.074179 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.074352 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.074394 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:05.076533 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:05.076535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:05.076535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:05.076542 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:05.077576 (+ 1034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:05.077900 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:05.077901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:05.077901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:05.077945 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:05.078112 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:05.078112 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:05.078113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:05.078206 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:05.078252 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:05.078445 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 14:49:05.078509 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:49:05.078742 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:05.070963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:40907)
0620 14:49:05.071091 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:05.071127 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:05.071694 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:05.071922 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:05.071926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:05.072480 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:49:05.072665 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:05.072671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.073332 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.073333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:05.074118 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:05.074121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.074427 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.074428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:05.074562 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:05.075022 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:49:05.075032 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:49:05.076460 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:49:05.077641 (+ 1181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:05.077646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:05.077648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:05.077850 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:05.077988 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:05.077988 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:05.077988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:05.078073 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:05.078247 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:49:05.078249 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:49:05.078356 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 14:49:05.078427 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:49:05.078546 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":60,"threads_started":1}
I20260620 14:49:05.079140 16221 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40171
I20260620 14:49:05.079229 16221 heartbeater.cc:464] Registering TS with master...
I20260620 14:49:05.079447 16221 heartbeater.cc:511] Master 127.0.98.62:40171 requested a full tablet report, sending...
I20260620 14:49:05.079804 15762 ts_manager.cc:194] Registered new tserver with Master: 57698b8e5272422781fd65d6f72b2619 (127.0.98.3:42949)
I20260620 14:49:05.080142 15762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40907
I20260620 14:49:05.086383 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:49:05.092885 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:05.087661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52528 (local address 127.0.98.62:40171)
0620 14:49:05.087700 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:05.087701 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:05.087823 (+ 122us) server_negotiation.cc:407] Connection header received
0620 14:49:05.087885 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:05.087886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:05.087896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:05.087942 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:05.088466 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.088939 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.089631 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.089821 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.089873 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:05.090869 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:05.090870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:05.090870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:05.090877 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:05.091911 (+ 1034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:05.092198 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:05.092199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:05.092199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:05.092250 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:05.092402 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:05.092403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:05.092403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:05.092539 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:05.092623 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:05.092722 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 14:49:05.092800 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:49:05.094072 15762 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:49:05.095635 15762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:49:05.101663 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:05.097755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58544 (local address 127.0.98.2:44095)
0620 14:49:05.097937 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:05.097942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:05.098251 (+ 309us) server_negotiation.cc:407] Connection header received
0620 14:49:05.098351 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:05.098354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:05.098386 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:05.098453 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:05.098526 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.099605 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.100836 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.101301 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.101363 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:05.101417 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:05.101534 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 14:49:05.101578 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
I20260620 14:49:05.101672 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:05.097612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43327 (local address 127.0.0.1:34496)
0620 14:49:05.097788 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:05.097826 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:05.097888 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:05.098265 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:05.098266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:05.098272 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:05.098329 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:05.098331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.099723 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.099724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:05.100581 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:05.100584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.101432 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.101433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:05.101482 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:05.101485 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:49:05.101511 (+ 26us) client_negotiation.cc:240] Negotiation successful
0620 14:49:05.101578 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":3,"thread_start_us":73,"threads_started":1}
I20260620 14:49:05.101676 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:05.097612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44095 (local address 127.0.0.1:58544)
0620 14:49:05.098192 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:05.098201 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:05.098279 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:05.098458 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:05.098458 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:05.098460 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:05.098490 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:05.098491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.099704 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.099705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:05.100765 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:05.100767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.101379 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.101380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:05.101439 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:05.101458 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:49:05.101520 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:49:05.101597 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":495,"thread_start_us":67,"threads_started":1}
I20260620 14:49:05.102035 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:05.097779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47064 (local address 127.0.98.3:42949)
0620 14:49:05.097998 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:05.098001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:05.098053 (+ 52us) server_negotiation.cc:407] Connection header received
0620 14:49:05.098116 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:05.098119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:05.098134 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:05.098218 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:05.098384 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.099118 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.100484 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.101190 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.101257 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:05.101314 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:05.101907 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 14:49:05.101953 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":82,"threads_started":1}
I20260620 14:49:05.102051 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:05.097942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34496 (local address 127.0.98.1:43327)
0620 14:49:05.098092 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:05.098096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:05.098104 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:49:05.098139 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:05.098141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:05.098154 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:05.098401 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:05.098472 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.099599 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.100678 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.101352 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.101849 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:05.101910 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:05.101934 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 14:49:05.101973 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260620 14:49:05.102083 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:05.097612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42949 (local address 127.0.0.1:47064)
0620 14:49:05.097988 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:05.097996 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:05.098054 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:05.098246 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:05.098249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:05.098252 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:05.098326 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:05.098328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.099233 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.099234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:05.100401 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:05.100403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.101351 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.101352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:05.101433 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:05.101447 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 14:49:05.101516 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:49:05.101580 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":32,"thread_start_us":65,"threads_started":1}
I20260620 14:49:05.102787 15887 tablet_service.cc:1467] Processing CreateTablet for tablet f37fcb80d0ca4e2da3a8586882a3af54 (DEFAULT_TABLE table=test-table [id=371429b937014fa6935e59a314cba01f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:49:05.102797 16021 tablet_service.cc:1467] Processing CreateTablet for tablet f37fcb80d0ca4e2da3a8586882a3af54 (DEFAULT_TABLE table=test-table [id=371429b937014fa6935e59a314cba01f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:49:05.102913 16155 tablet_service.cc:1467] Processing CreateTablet for tablet f37fcb80d0ca4e2da3a8586882a3af54 (DEFAULT_TABLE table=test-table [id=371429b937014fa6935e59a314cba01f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:49:05.103106 15887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f37fcb80d0ca4e2da3a8586882a3af54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:49:05.103111 16021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f37fcb80d0ca4e2da3a8586882a3af54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:49:05.103188 16155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f37fcb80d0ca4e2da3a8586882a3af54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:49:05.105576 16235 tablet_bootstrap.cc:492] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f: Bootstrap starting.
I20260620 14:49:05.106199 16235 tablet_bootstrap.cc:654] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f: Neither blocks nor log segments found. Creating new log.
I20260620 14:49:05.106218 16236 tablet_bootstrap.cc:492] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d: Bootstrap starting.
I20260620 14:49:05.106875 16236 tablet_bootstrap.cc:654] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d: Neither blocks nor log segments found. Creating new log.
I20260620 14:49:05.106863 16235 log.cc:824] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:05.107396 16236 log.cc:824] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:05.107560 16237 tablet_bootstrap.cc:492] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619: Bootstrap starting.
I20260620 14:49:05.107606 16235 tablet_bootstrap.cc:492] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f: No bootstrap required, opened a new log
I20260620 14:49:05.107703 16235 ts_tablet_manager.cc:1397] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:49:05.108136 16237 tablet_bootstrap.cc:654] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619: Neither blocks nor log segments found. Creating new log.
I20260620 14:49:05.108179 16236 tablet_bootstrap.cc:492] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d: No bootstrap required, opened a new log
I20260620 14:49:05.108232 16236 ts_tablet_manager.cc:1397] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:49:05.108572 16237 log.cc:824] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:05.109191 16237 tablet_bootstrap.cc:492] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619: No bootstrap required, opened a new log
I20260620 14:49:05.109279 16237 ts_tablet_manager.cc:1397] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:49:05.109658 16235 raft_consensus.cc:348] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } } peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } }
I20260620 14:49:05.109849 16235 raft_consensus.cc:374] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:49:05.109885 16235 raft_consensus.cc:729] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce64d8a46f8f4dd0910203fc20d0646f, State: Initialized, Role: FOLLOWER
I20260620 14:49:05.110069 16235 consensus_queue.cc:260] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } } peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } }
I20260620 14:49:05.110098 16236 raft_consensus.cc:348] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } } peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } }
I20260620 14:49:05.110284 16236 raft_consensus.cc:374] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:49:05.110370 16236 raft_consensus.cc:729] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d60b6c9420004fd488ec636183caea7d, State: Initialized, Role: FOLLOWER
I20260620 14:49:05.110446 15953 heartbeater.cc:503] Master 127.0.98.62:40171 was elected leader, sending a full tablet report...
I20260620 14:49:05.110561 16236 consensus_queue.cc:260] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } } peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } }
I20260620 14:49:05.110440 16235 ts_tablet_manager.cc:1428] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:49:05.110831 16236 ts_tablet_manager.cc:1428] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 14:49:05.110946 16087 heartbeater.cc:503] Master 127.0.98.62:40171 was elected leader, sending a full tablet report...
I20260620 14:49:05.110929 16237 raft_consensus.cc:348] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } } peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } }
I20260620 14:49:05.111274 16237 raft_consensus.cc:374] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:49:05.111346 16237 raft_consensus.cc:729] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57698b8e5272422781fd65d6f72b2619, State: Initialized, Role: FOLLOWER
I20260620 14:49:05.111469 16237 consensus_queue.cc:260] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } } peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } }
I20260620 14:49:05.111756 16237 ts_tablet_manager.cc:1428] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:49:05.111827 16221 heartbeater.cc:503] Master 127.0.98.62:40171 was elected leader, sending a full tablet report...
I20260620 14:49:05.123311 16241 raft_consensus.cc:482] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:49:05.123406 16241 raft_consensus.cc:504] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } } peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } }
I20260620 14:49:05.123728 16241 leader_election.cc:288] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57698b8e5272422781fd65d6f72b2619 (127.0.98.3:42949), d60b6c9420004fd488ec636183caea7d (127.0.98.2:44095)
I20260620 14:49:05.127315 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:05.123905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42993 (local address 127.0.98.3:42949)
0620 14:49:05.123942 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:05.123945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:05.123974 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:49:05.124058 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:05.124059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:05.124068 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:05.124124 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:05.124260 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.124985 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.126420 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.126983 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.127039 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:05.127048 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:05.127211 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:49:05.127248 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:49:05.127315 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:05.123915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45609 (local address 127.0.98.2:44095)
0620 14:49:05.123956 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:05.123958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:05.124103 (+ 145us) server_negotiation.cc:407] Connection header received
0620 14:49:05.124144 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:05.124146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:05.124167 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:05.124206 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:05.124323 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.124994 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.126420 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.126977 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.127039 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:05.127048 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:05.127211 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:49:05.127248 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:49:05.127341 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:05.123880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42949 (local address 127.0.98.1:42993)
0620 14:49:05.123934 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:05.123945 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:05.123990 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:05.124133 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:05.124134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:05.124136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:05.124200 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:05.124202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.125051 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.125052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:05.126340 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:05.126343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.127054 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.127054 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:05.127149 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:05.127153 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:49:05.127201 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:49:05.127255 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":45}
I20260620 14:49:05.127347 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:05.123882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44095 (local address 127.0.98.1:45609)
0620 14:49:05.124041 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:05.124050 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:05.124095 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:05.124215 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:05.124216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:05.124217 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:05.124269 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:05.124270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.125051 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.125052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:05.126340 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:05.126343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.127054 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.127055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:05.127149 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:05.127153 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:49:05.127201 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:49:05.127248 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":85,"thread_start_us":102,"threads_started":1}
I20260620 14:49:05.127743 16041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37fcb80d0ca4e2da3a8586882a3af54" candidate_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d60b6c9420004fd488ec636183caea7d" is_pre_election: true
I20260620 14:49:05.127780 16175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37fcb80d0ca4e2da3a8586882a3af54" candidate_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57698b8e5272422781fd65d6f72b2619" is_pre_election: true
I20260620 14:49:05.127861 16041 raft_consensus.cc:2454] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce64d8a46f8f4dd0910203fc20d0646f in term 0.
I20260620 14:49:05.127902 16175 raft_consensus.cc:2454] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce64d8a46f8f4dd0910203fc20d0646f in term 0.
I20260620 14:49:05.128073 15843 leader_election.cc:302] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce64d8a46f8f4dd0910203fc20d0646f, d60b6c9420004fd488ec636183caea7d; no voters:
I20260620 14:49:05.128219 16241 raft_consensus.cc:2781] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:49:05.128283 16241 raft_consensus.cc:482] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:49:05.128304 16241 raft_consensus.cc:3037] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:49:05.129000 16241 raft_consensus.cc:504] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } } peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } }
I20260620 14:49:05.129112 16241 leader_election.cc:288] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 1 election: Requested vote from peers 57698b8e5272422781fd65d6f72b2619 (127.0.98.3:42949), d60b6c9420004fd488ec636183caea7d (127.0.98.2:44095)
I20260620 14:49:05.129251 16041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37fcb80d0ca4e2da3a8586882a3af54" candidate_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d60b6c9420004fd488ec636183caea7d"
I20260620 14:49:05.129261 16175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37fcb80d0ca4e2da3a8586882a3af54" candidate_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57698b8e5272422781fd65d6f72b2619"
I20260620 14:49:05.129321 16175 raft_consensus.cc:3037] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:49:05.129324 16041 raft_consensus.cc:3037] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:49:05.129957 16175 raft_consensus.cc:2454] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce64d8a46f8f4dd0910203fc20d0646f in term 1.
I20260620 14:49:05.129988 16041 raft_consensus.cc:2454] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce64d8a46f8f4dd0910203fc20d0646f in term 1.
I20260620 14:49:05.130117 15840 leader_election.cc:302] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57698b8e5272422781fd65d6f72b2619, ce64d8a46f8f4dd0910203fc20d0646f; no voters:
I20260620 14:49:05.130244 16241 raft_consensus.cc:2781] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:49:05.130369 16241 raft_consensus.cc:686] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 LEADER]: Becoming Leader. State: Replica: ce64d8a46f8f4dd0910203fc20d0646f, State: Running, Role: LEADER
I20260620 14:49:05.130451 16241 consensus_queue.cc:237] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } } peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } }
I20260620 14:49:05.131170 15761 catalog_manager.cc:5495] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f reported cstate change: term changed from 0 to 1, leader changed from <none> to ce64d8a46f8f4dd0910203fc20d0646f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } health_report { overall_health: HEALTHY } } }
I20260620 14:49:05.140231 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:05.137965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34510 (local address 127.0.98.1:43327)
0620 14:49:05.138009 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:05.138011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:05.138020 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:49:05.138055 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:05.138055 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:05.138076 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:05.138137 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:49:05.138262 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.138719 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.139512 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:05.139719 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:05.139781 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:05.139929 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:49:05.140029 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:49:05.140098 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:05.140118 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 14:49:05.140138 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:49:05.141842 16175 raft_consensus.cc:1261] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619 [term 1 FOLLOWER]: Refusing update from remote peer ce64d8a46f8f4dd0910203fc20d0646f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:49:05.141850 16041 raft_consensus.cc:1261] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d [term 1 FOLLOWER]: Refusing update from remote peer ce64d8a46f8f4dd0910203fc20d0646f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:49:05.142179 16241 consensus_queue.cc:1035] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:49:05.142306 16245 consensus_queue.cc:1035] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:49:05.144603 16248 mvcc.cc:205] Tried to move back new op lower bound from 7298936607298646016 to 7298936607254790144. Current Snapshot: MvccSnapshot[applied={T|T < 7298936607298646016}]
I20260620 14:49:05.145035 16250 mvcc.cc:205] Tried to move back new op lower bound from 7298936607298646016 to 7298936607254790144. Current Snapshot: MvccSnapshot[applied={T|T < 7298936607298646016}]
I20260620 14:49:05.145290 16249 mvcc.cc:205] Tried to move back new op lower bound from 7298936607298646016 to 7298936607254790144. Current Snapshot: MvccSnapshot[applied={T|T < 7298936607298646016}]
I20260620 14:49:05.146842 15762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:49:05.146939 15762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:49:05.147684 15762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: Sending DeleteTablet for 3 replicas of tablet f37fcb80d0ca4e2da3a8586882a3af54
I20260620 14:49:05.147982 15887 tablet_service.cc:1514] Processing DeleteTablet for tablet f37fcb80d0ca4e2da3a8586882a3af54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:49:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34496
I20260620 14:49:05.148026 16155 tablet_service.cc:1514] Processing DeleteTablet for tablet f37fcb80d0ca4e2da3a8586882a3af54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:49:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47064
I20260620 14:49:05.148177 16258 tablet_replica.cc:330] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f: stopping tablet replica
I20260620 14:49:05.148269 16258 raft_consensus.cc:2229] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:49:05.148242 16021 tablet_service.cc:1514] Processing DeleteTablet for tablet f37fcb80d0ca4e2da3a8586882a3af54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:49:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58544
I20260620 14:49:05.148361 16258 raft_consensus.cc:2258] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:49:05.148437 16260 tablet_replica.cc:330] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d: stopping tablet replica
I20260620 14:49:05.148510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 15824
I20260620 14:49:05.148552 16260 raft_consensus.cc:2229] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:49:05.148635 16260 raft_consensus.cc:2258] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:49:05.149228 16260 ts_tablet_manager.cc:1905] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:49:05.150596 16260 ts_tablet_manager.cc:1918] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:49:05.150641 16260 log.cc:1196] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/f37fcb80d0ca4e2da3a8586882a3af54
I20260620 14:49:05.150817 16260 ts_tablet_manager.cc:1939] T f37fcb80d0ca4e2da3a8586882a3af54 P d60b6c9420004fd488ec636183caea7d: Deleting consensus metadata
I20260620 14:49:05.151160 15749 catalog_manager.cc:4841] TS d60b6c9420004fd488ec636183caea7d (127.0.98.2:44095): tablet f37fcb80d0ca4e2da3a8586882a3af54 (table test-table [id=371429b937014fa6935e59a314cba01f]) successfully deleted
W20260620 14:49:05.153494 15747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43327 (error 108)
I20260620 14:49:05.153702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 15958
I20260620 14:49:05.153803 16259 tablet_replica.cc:330] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619: stopping tablet replica
I20260620 14:49:05.153895 16259 raft_consensus.cc:2229] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:49:05.154266 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:05.153928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43327 (local address 127.0.0.1:34518)
0620 14:49:05.154175 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:05.154219 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223}
W20260620 14:49:05.154376 15747 catalog_manager.cc:4584] TS ce64d8a46f8f4dd0910203fc20d0646f (127.0.98.1:43327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f37fcb80d0ca4e2da3a8586882a3af54: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43327: connect: Connection refused (error 111)
I20260620 14:49:05.154482 16259 raft_consensus.cc:2258] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:49:05.158636 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 16092
I20260620 14:49:05.158833 16259 ts_tablet_manager.cc:1905] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:49:05.162292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 15730
I20260620 14:49:05.162444 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:05.162325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42949 (local address 127.0.0.1:47076)
0620 14:49:05.162379 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:05.162411 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:42949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:49:25.166687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40171
--webserver_interface=127.0.98.62
--webserver_port=37505
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:49:25.245690 16262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:25.245904 16262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:25.245934 16262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:25.247061 16262 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:49:25.247098 16262 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:49:25.247114 16262 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:49:25.247129 16262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:49:25.247145 16262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:25.247164 16262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:25.247186 16262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:49:25.247203 16262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:49:25.248693 16262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:49:25.248940 16262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:25.249135 16262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:25.257149 16269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:25.257180 16262 server_base.cc:1034] running on GCE node
W20260620 14:49:25.257262 16268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:25.257231 16271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:25.257586 16262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:25.257869 16262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:25.258999 16262 hybrid_clock.cc:648] HybridClock initialized: now 1781966965258986 us; error 34 us; skew 500 ppm
Jun 20 14:49:25 dist-test-slave-cx89 krb5kdc[15717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:25.261361 16262 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:49:25.261765 16262 webserver.cc:460] Webserver started at http://127.0.98.62:37505/ using document root <none> and password file <none>
I20260620 14:49:25.261978 16262 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:25.262032 16262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:25.263360 16262 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:49:25.263962 16278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:25.264103 16262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:25.264185 16262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "080535b714844b028b54559fceca82a0"
format_stamp: "Formatted at 2026-06-20 14:49:04 on dist-test-slave-cx89"
server_key: "45ece8f915c105a9d37fa7d7fa836567"
server_key_iv: "d8d04417ab01d4e6cbb15a0287bd8c51"
server_key_version: "encryptionkey@0"
I20260620 14:49:25.264565 16262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:25.278568 16262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:25.280429 16262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:25.280560 16262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:25.284368 16262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40171
I20260620 14:49:25.284400 16330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40171 every 8 connection(s)
I20260620 14:49:25.284745 16262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:49:25.286118 16331 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:49:25.286794 16331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: Bootstrap starting.
I20260620 14:49:25.291251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 16262
I20260620 14:49:25.291538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43327
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40171
--webserver_port=43173
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:49:25.294843 16331 log.cc:824] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:25.296905 16331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:49:25.297159 16331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: Bootstrap complete.
I20260620 14:49:25.299233 16331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080535b714844b028b54559fceca82a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40171 } }
I20260620 14:49:25.299505 16331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 080535b714844b028b54559fceca82a0, State: Initialized, Role: FOLLOWER
I20260620 14:49:25.299664 16331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080535b714844b028b54559fceca82a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40171 } }
I20260620 14:49:25.299721 16331 raft_consensus.cc:388] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:49:25.299769 16331 raft_consensus.cc:482] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:49:25.299876 16331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:49:25.300953 16331 raft_consensus.cc:504] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080535b714844b028b54559fceca82a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40171 } }
I20260620 14:49:25.301049 16331 leader_election.cc:302] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 080535b714844b028b54559fceca82a0; no voters:
I20260620 14:49:25.301203 16331 leader_election.cc:288] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:49:25.301252 16335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:49:25.301373 16335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 2 LEADER]: Becoming Leader. State: Replica: 080535b714844b028b54559fceca82a0, State: Running, Role: LEADER
I20260620 14:49:25.301471 16331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:49:25.301519 16335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080535b714844b028b54559fceca82a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40171 } }
I20260620 14:49:25.301983 16336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "080535b714844b028b54559fceca82a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080535b714844b028b54559fceca82a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40171 } } }
I20260620 14:49:25.302070 16336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:49:25.302273 16337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 080535b714844b028b54559fceca82a0. Latest consensus state: current_term: 2 leader_uuid: "080535b714844b028b54559fceca82a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080535b714844b028b54559fceca82a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40171 } } }
I20260620 14:49:25.302335 16337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:49:25.303164 16351 catalog_manager.cc:1259] Loaded cluster ID: 3326b935460342819d44e6a40743ed1f
I20260620 14:49:25.303212 16351 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: loading cluster ID for follower catalog manager: success
I20260620 14:49:25.304364 16351 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: acquiring CA information for follower catalog manager: success
I20260620 14:49:25.304855 16351 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:49:25.304929 16345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:49:25.305303 16345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:49:25.305372 16345 catalog_manager.cc:1259] Loaded cluster ID: 3326b935460342819d44e6a40743ed1f
I20260620 14:49:25.305394 16345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:49:25.305545 16345 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:49:25.305611 16345 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: Loaded TSK: 0
I20260620 14:49:25.325117 16345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: Generated new TSK 1
I20260620 14:49:25.325232 16345 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:49:25.371148 16333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:25.371356 16333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:25.371387 16333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:25.372534 16333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:25.372575 16333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:25.372611 16333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:49:25.374018 16333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43173
--tserver_master_addrs=127.0.98.62:40171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:49:25.374241 16333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:25.374452 16333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:25.379127 16358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:25.379117 16359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:25.379117 16361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:25.379304 16333 server_base.cc:1034] running on GCE node
I20260620 14:49:25.379578 16333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:25.379798 16333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:25.380951 16333 hybrid_clock.cc:648] HybridClock initialized: now 1781966965380938 us; error 31 us; skew 500 ppm
Jun 20 14:49:25 dist-test-slave-cx89 krb5kdc[15717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:25.383385 16333 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:49:25.383836 16333 webserver.cc:460] Webserver started at http://127.0.98.1:43173/ using document root <none> and password file <none>
I20260620 14:49:25.384057 16333 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:25.384109 16333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:25.385368 16333 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:49:25.385969 16368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:25.386125 16333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:49:25.386229 16333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "ce64d8a46f8f4dd0910203fc20d0646f"
format_stamp: "Formatted at 2026-06-20 14:49:04 on dist-test-slave-cx89"
server_key: "3d13e8c51f583dab4c4bad25c01dfbfc"
server_key_iv: "0b43c15d5d092d0abecf103a0750949c"
server_key_version: "encryptionkey@0"
I20260620 14:49:25.386662 16333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:25.406011 16333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:25.408438 16333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:25.408591 16333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:25.408849 16333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:49:25.409320 16375 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:49:25.410697 16333 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:49:25.410759 16333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:49:25.410810 16333 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:49:25.411681 16333 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:49:25.411725 16333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:49:25.411916 16375 tablet_bootstrap.cc:492] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f: Bootstrap starting.
I20260620 14:49:25.418778 16333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43327
I20260620 14:49:25.419224 16333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:49:25.421592 16375 log.cc:824] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:25.423609 16375 tablet_bootstrap.cc:492] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:49:25.423836 16375 tablet_bootstrap.cc:492] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f: Bootstrap complete.
I20260620 14:49:25.424549 16375 ts_tablet_manager.cc:1397] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.010s
I20260620 14:49:25.425717 16375 raft_consensus.cc:348] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } } peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } }
I20260620 14:49:25.425956 16375 raft_consensus.cc:729] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce64d8a46f8f4dd0910203fc20d0646f, State: Initialized, Role: FOLLOWER
I20260620 14:49:25.426153 16375 consensus_queue.cc:260] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } } peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } }
I20260620 14:49:25.426239 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 16333
I20260620 14:49:25.426313 16482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43327 every 8 connection(s)
I20260620 14:49:25.426445 16375 ts_tablet_manager.cc:1428] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 14:49:25.426494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44095
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40171
--webserver_port=45453
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:49:25 dist-test-slave-cx89 krb5kdc[15717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:49:25.438419 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:25.420118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:58497)
0620 14:49:25.426693 (+ 6575us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:25.426732 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:25.426922 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:25.427356 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:25.427360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:25.427735 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:49:25.428008 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:25.428014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.429620 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.429621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:25.430910 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:25.430912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.431764 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.431765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:25.431912 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:25.432514 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:49:25.432524 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:49:25.434418 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:49:25.436657 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:25.436664 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:25.436667 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:25.436972 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:25.437257 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:25.437258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:25.437258 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:25.437383 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:25.437778 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:49:25.437782 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:49:25.437940 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 14:49:25.438044 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:49:25.438208 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6533,"thread_start_us":95,"threads_started":1}
I20260620 14:49:25.438742 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:25.420234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58497 (local address 127.0.98.62:40171)
0620 14:49:25.420427 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:25.420437 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:25.426889 (+ 6452us) server_negotiation.cc:407] Connection header received
0620 14:49:25.427107 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:25.427113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:25.427172 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:25.427313 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:25.428142 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.429505 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.431010 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.431676 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.431793 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:25.434551 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:25.434557 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:25.434557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:25.434570 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:25.436569 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:25.437069 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:25.437073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:25.437073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:25.437185 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:25.437442 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:25.437442 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:25.437443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:25.437717 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:25.437793 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:25.438127 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 14:49:25.438265 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260620 14:49:25.439289 16483 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40171
I20260620 14:49:25.439393 16483 heartbeater.cc:464] Registering TS with master...
I20260620 14:49:25.439587 16483 heartbeater.cc:511] Master 127.0.98.62:40171 requested a full tablet report, sending...
I20260620 14:49:25.440332 16295 ts_manager.cc:194] Registered new tserver with Master: ce64d8a46f8f4dd0910203fc20d0646f (127.0.98.1:43327)
I20260620 14:49:25.441545 16295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58497
I20260620 14:49:25.442453 16483 heartbeater.cc:503] Master 127.0.98.62:40171 was elected leader, sending a full tablet report...
Jun 20 14:49:25 dist-test-slave-cx89 krb5kdc[15717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:49:25.449549 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:25.440937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43327 (local address 127.0.0.1:50248)
0620 14:49:25.441331 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:25.441365 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:25.441426 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:25.441808 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:25.441815 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:25.442131 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:49:25.442186 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:25.442188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.443526 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.443527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:25.444261 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:25.444264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.444853 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.444856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:25.444925 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:25.445402 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:49:25.445407 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:49:25.446898 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:49:25.448625 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:25.448631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:25.448632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:25.448809 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:25.448992 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:25.448992 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:25.448993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:25.449038 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:25.449300 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:49:25.449303 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:49:25.449342 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:49:25.449399 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 14:49:25.449483 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":90,"threads_started":1}
I20260620 14:49:25.449736 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:25.441179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50248 (local address 127.0.98.1:43327)
0620 14:49:25.441547 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:25.441551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:25.441560 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:49:25.441591 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:25.441593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:25.441681 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:25.441744 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:25.442268 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.443461 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.444342 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.444782 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.444856 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:25.446980 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:25.446988 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:25.446988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:25.446996 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:25.448555 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:25.448875 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:25.448878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:25.448878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:25.448938 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:25.449084 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:25.449084 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:25.449085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:25.449245 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:25.449312 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:25.449439 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 14:49:25.449529 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":75,"threads_started":1}
I20260620 14:49:25.450062 16402 tablet_service.cc:1514] Processing DeleteTablet for tablet f37fcb80d0ca4e2da3a8586882a3af54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:49:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50248
I20260620 14:49:25.450546 16492 tablet_replica.cc:330] stopping tablet replica
I20260620 14:49:25.450632 16492 raft_consensus.cc:2229] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:49:25.450698 16492 raft_consensus.cc:2258] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:49:25.450968 16492 ts_tablet_manager.cc:1905] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:49:25.453114 16492 ts_tablet_manager.cc:1918] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:49:25.453172 16492 log.cc:1196] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/wal/wals/f37fcb80d0ca4e2da3a8586882a3af54
I20260620 14:49:25.453349 16492 ts_tablet_manager.cc:1939] T f37fcb80d0ca4e2da3a8586882a3af54 P ce64d8a46f8f4dd0910203fc20d0646f: Deleting consensus metadata
I20260620 14:49:25.453794 16280 catalog_manager.cc:4841] TS ce64d8a46f8f4dd0910203fc20d0646f (127.0.98.1:43327): tablet f37fcb80d0ca4e2da3a8586882a3af54 (table test-table [id=371429b937014fa6935e59a314cba01f]) successfully deleted
W20260620 14:49:25.509670 16489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:25.509874 16489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:25.509907 16489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:25.511040 16489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:25.511078 16489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:25.511112 16489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:49:25.512631 16489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45453
--tserver_master_addrs=127.0.98.62:40171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:49:25.512858 16489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:25.513064 16489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:25.517664 16498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:25.517688 16500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:25.517688 16497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:25.517781 16489 server_base.cc:1034] running on GCE node
I20260620 14:49:25.518007 16489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:25.518246 16489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:25.519388 16489 hybrid_clock.cc:648] HybridClock initialized: now 1781966965519373 us; error 27 us; skew 500 ppm
Jun 20 14:49:25 dist-test-slave-cx89 krb5kdc[15717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:25.521693 16489 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:49:25.522092 16489 webserver.cc:460] Webserver started at http://127.0.98.2:45453/ using document root <none> and password file <none>
I20260620 14:49:25.522274 16489 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:25.522329 16489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:25.523514 16489 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:49:25.523981 16507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:25.524123 16489 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:25.524185 16489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "d60b6c9420004fd488ec636183caea7d"
format_stamp: "Formatted at 2026-06-20 14:49:04 on dist-test-slave-cx89"
server_key: "40a102b1de1284eaed21a42280eecc7d"
server_key_iv: "c756c9e3c1bc91da32ce119fcdff8f07"
server_key_version: "encryptionkey@0"
I20260620 14:49:25.524523 16489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:25.548710 16489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:25.550443 16489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:25.550559 16489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:25.550781 16489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:49:25.551059 16489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:49:25.551090 16489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:25.551118 16489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:49:25.551136 16489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:25.556354 16489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44095
I20260620 14:49:25.556401 16620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44095 every 8 connection(s)
I20260620 14:49:25.556721 16489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:49:25.561657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 16489
I20260620 14:49:25.561959 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42949
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40171
--webserver_port=42277
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:49:25 dist-test-slave-cx89 krb5kdc[15717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:49:25.565785 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:25.557493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48299 (local address 127.0.98.62:40171)
0620 14:49:25.557546 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:25.557548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:25.557755 (+ 207us) server_negotiation.cc:407] Connection header received
0620 14:49:25.558228 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:25.558230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:25.558244 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:25.558297 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:25.559198 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.559748 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.560691 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.560886 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.560937 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:25.563260 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:25.563262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:25.563262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:25.563270 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:25.564523 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:25.564937 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:25.564938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:25.564938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:25.564987 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:25.565202 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:25.565203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:25.565204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:25.565307 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:25.565363 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:25.565611 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 14:49:25.565697 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:49:25.565961 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:25.557489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:48299)
0620 14:49:25.557629 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:25.557663 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:25.558126 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:25.558364 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:25.558367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:25.558914 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:49:25.559107 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:25.559112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.559837 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.559838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:25.560627 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:25.560629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.560979 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.560980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:25.561103 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:25.561604 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:49:25.561616 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:49:25.563174 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:49:25.564596 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:25.564603 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:25.564605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:25.564863 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:25.565037 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:25.565038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:25.565038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:25.565141 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:25.565361 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:49:25.565367 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:49:25.565500 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 14:49:25.565586 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:49:25.565749 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260620 14:49:25.566406 16621 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40171
I20260620 14:49:25.566490 16621 heartbeater.cc:464] Registering TS with master...
I20260620 14:49:25.566655 16621 heartbeater.cc:511] Master 127.0.98.62:40171 requested a full tablet report, sending...
I20260620 14:49:25.567025 16295 ts_manager.cc:194] Registered new tserver with Master: d60b6c9420004fd488ec636183caea7d (127.0.98.2:44095)
I20260620 14:49:25.567435 16295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48299
W20260620 14:49:25.640329 16624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:25.640537 16624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:25.640566 16624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:25.641695 16624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:25.641736 16624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:25.641768 16624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:49:25.643152 16624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42277
--tserver_master_addrs=127.0.98.62:40171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:49:25.643374 16624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:25.643597 16624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:25.648236 16632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:25.648340 16630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:25.648240 16629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:25.648398 16624 server_base.cc:1034] running on GCE node
I20260620 14:49:25.648591 16624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:25.648814 16624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:25.649968 16624 hybrid_clock.cc:648] HybridClock initialized: now 1781966965649955 us; error 29 us; skew 500 ppm
Jun 20 14:49:25 dist-test-slave-cx89 krb5kdc[15717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:25.652254 16624 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:49:25.652642 16624 webserver.cc:460] Webserver started at http://127.0.98.3:42277/ using document root <none> and password file <none>
I20260620 14:49:25.652838 16624 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:25.652899 16624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:25.654138 16624 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:49:25.654745 16639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:25.654892 16624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:49:25.654970 16624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "57698b8e5272422781fd65d6f72b2619"
format_stamp: "Formatted at 2026-06-20 14:49:05 on dist-test-slave-cx89"
server_key: "82269baba8eeed589205155af8d5dfea"
server_key_iv: "58640e68c623ffd2da51e2321986c11a"
server_key_version: "encryptionkey@0"
I20260620 14:49:25.655280 16624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:25.666386 16624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:25.668440 16624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:25.668572 16624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:25.668776 16624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:49:25.669158 16646 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:49:25.670557 16624 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:49:25.670596 16624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:49:25.670635 16624 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:49:25.671232 16624 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:49:25.671265 16624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:49:25.671344 16646 tablet_bootstrap.cc:492] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619: Bootstrap starting.
I20260620 14:49:25.673007 16646 log.cc:824] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:25.676172 16646 tablet_bootstrap.cc:492] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:49:25.676510 16646 tablet_bootstrap.cc:492] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619: Bootstrap complete.
I20260620 14:49:25.676671 16646 ts_tablet_manager.cc:1397] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260620 14:49:25.677861 16624 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42949
I20260620 14:49:25.677896 16754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42949 every 8 connection(s)
I20260620 14:49:25.678221 16624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:49:25.678483 16646 raft_consensus.cc:348] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } } peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } }
I20260620 14:49:25.678848 16646 raft_consensus.cc:729] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57698b8e5272422781fd65d6f72b2619, State: Initialized, Role: FOLLOWER
I20260620 14:49:25.679006 16646 consensus_queue.cc:260] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } } peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } }
I20260620 14:49:25.679252 16646 ts_tablet_manager.cc:1428] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
Jun 20 14:49:25 dist-test-slave-cx89 krb5kdc[15717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:49:25.686813 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:25.678908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38517 (local address 127.0.98.62:40171)
0620 14:49:25.678956 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:25.678958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:25.679174 (+ 216us) server_negotiation.cc:407] Connection header received
0620 14:49:25.679344 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:25.679346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:25.679358 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:25.679434 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:25.680031 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.680510 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.681472 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.681662 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.681716 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:25.684166 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:25.684167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:25.684167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:25.684174 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:25.685253 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:25.685714 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:25.685716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:25.685717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:25.685780 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:25.686039 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:25.686040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:25.686041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:25.686158 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:25.686215 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:25.686552 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 14:49:25.686666 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:49:25.686801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 16624
I20260620 14:49:25.686964 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:25.678914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:38517)
0620 14:49:25.679056 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:25.679096 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:25.679264 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:25.679502 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:25.679506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:25.679753 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:49:25.679939 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:25.679944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.680609 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.680610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:25.681390 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:25.681392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.681725 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.681726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:25.681843 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:25.682341 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:49:25.682349 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:49:25.684072 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:49:25.685337 (+ 1265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:25.685345 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:25.685347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:25.685630 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:25.685842 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:25.685843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:25.685843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:25.685979 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:25.686212 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:49:25.686217 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:49:25.686398 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 14:49:25.686503 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 14:49:25.686660 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260620 14:49:25.687724 16755 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40171
I20260620 14:49:25.687858 16755 heartbeater.cc:464] Registering TS with master...
I20260620 14:49:25.688066 16755 heartbeater.cc:511] Master 127.0.98.62:40171 requested a full tablet report, sending...
I20260620 14:49:25.688619 16295 ts_manager.cc:194] Registered new tserver with Master: 57698b8e5272422781fd65d6f72b2619 (127.0.98.3:42949)
I20260620 14:49:25.689177 16295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38517
I20260620 14:49:25.689934 16755 heartbeater.cc:503] Master 127.0.98.62:40171 was elected leader, sending a full tablet report...
Jun 20 14:49:25 dist-test-slave-cx89 krb5kdc[15717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260620 14:49:25.694268 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:25.687349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40412 (local address 127.0.98.62:40171)
0620 14:49:25.687735 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:25.687736 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:25.687953 (+ 217us) server_negotiation.cc:407] Connection header received
0620 14:49:25.688067 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:25.688068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:25.688078 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:25.688166 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:25.688770 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.689408 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.690349 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.690578 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.690610 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:25.692011 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:25.692012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:25.692012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:25.692018 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:25.693296 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:25.693604 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:25.693605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:25.693606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:25.693660 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:25.693820 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:25.693821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:25.693822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:25.693924 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:25.694075 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:25.694109 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 14:49:25.694180 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363}
I20260620 14:49:25.695005 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:49:25.696251 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:25.688896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42949 (local address 127.0.0.1:50548)
0620 14:49:25.688940 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:25.688949 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:25.688980 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:25.689251 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:25.689252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:25.689402 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:49:25.689452 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:25.689453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.690671 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.690672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:25.691481 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:25.691484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.691961 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.691964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:25.692030 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:25.692351 (+ 321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:49:25.692354 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:49:25.693519 (+ 1165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:49:25.695248 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:25.695249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:25.695250 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:25.695470 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:25.695645 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:25.695645 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:25.695646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:25.695699 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:25.695949 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:49:25.695950 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:49:25.695991 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:49:25.696055 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:49:25.696157 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 14:49:25.696269 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:25.688969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.0.98.3:42949)
0620 14:49:25.689112 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:25.689116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:25.689124 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:49:25.689155 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:25.689157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:25.689170 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:25.689352 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:25.689524 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.690593 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.691556 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.691893 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.691945 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:25.693587 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:25.693594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:25.693594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:25.693601 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:25.695180 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:25.695529 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:25.695532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:25.695533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:25.695589 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:25.695749 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:25.695750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:25.695751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:25.695889 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:25.695957 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:25.696089 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 14:49:25.696169 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260620 14:49:25.696607 16689 tablet_service.cc:1514] Processing DeleteTablet for tablet f37fcb80d0ca4e2da3a8586882a3af54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:49:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50548
I20260620 14:49:25.697122 16762 tablet_replica.cc:330] stopping tablet replica
I20260620 14:49:25.697237 16762 raft_consensus.cc:2229] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:49:25.697314 16762 raft_consensus.cc:2258] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:49:25.697610 16762 ts_tablet_manager.cc:1905] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:49:25.698033 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:25.695319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40414 (local address 127.0.98.62:40171)
0620 14:49:25.695367 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:25.695369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:25.695471 (+ 102us) server_negotiation.cc:407] Connection header received
0620 14:49:25.695550 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:25.695551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:25.695585 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:25.695645 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:49:25.695742 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.696269 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.697280 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.697550 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.697602 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:25.697775 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:49:25.697868 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:49:25.697920 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:25.697982 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 14:49:25.697994 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:49:25.699832 16762 ts_tablet_manager.cc:1918] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:49:25.699905 16762 log.cc:1196] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/f37fcb80d0ca4e2da3a8586882a3af54
I20260620 14:49:25.700117 16762 ts_tablet_manager.cc:1939] T f37fcb80d0ca4e2da3a8586882a3af54 P 57698b8e5272422781fd65d6f72b2619: Deleting consensus metadata
I20260620 14:49:25.698256 16295 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:49:25.700462 16295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:49:25.700591 16279 catalog_manager.cc:4841] TS 57698b8e5272422781fd65d6f72b2619 (127.0.98.3:42949): tablet f37fcb80d0ca4e2da3a8586882a3af54 (table test-table [id=371429b937014fa6935e59a314cba01f]) successfully deleted
I20260620 14:49:25.702251 16689 tablet_service.cc:1467] Processing CreateTablet for tablet 06b424d310434fae80da1628fdf6d0b6 (DEFAULT_TABLE table=test-table [id=c38557672a0545678bcf48b6bf1cc161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:49:25.702302 16402 tablet_service.cc:1467] Processing CreateTablet for tablet 06b424d310434fae80da1628fdf6d0b6 (DEFAULT_TABLE table=test-table [id=c38557672a0545678bcf48b6bf1cc161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:49:25.702410 16689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b424d310434fae80da1628fdf6d0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:49:25.702478 16402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b424d310434fae80da1628fdf6d0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:49:25.703626 16646 tablet_bootstrap.cc:492] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619: Bootstrap starting.
I20260620 14:49:25.703776 16375 tablet_bootstrap.cc:492] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f: Bootstrap starting.
I20260620 14:49:25.703981 16646 tablet_bootstrap.cc:654] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619: Neither blocks nor log segments found. Creating new log.
I20260620 14:49:25.704044 16375 tablet_bootstrap.cc:654] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f: Neither blocks nor log segments found. Creating new log.
I20260620 14:49:25.704509 16646 tablet_bootstrap.cc:492] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619: No bootstrap required, opened a new log
I20260620 14:49:25.704515 16375 tablet_bootstrap.cc:492] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f: No bootstrap required, opened a new log
I20260620 14:49:25.704551 16375 ts_tablet_manager.cc:1397] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:49:25.704551 16646 ts_tablet_manager.cc:1397] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:49:25.704821 16646 raft_consensus.cc:348] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } }
I20260620 14:49:25.704885 16646 raft_consensus.cc:374] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:49:25.704922 16646 raft_consensus.cc:729] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57698b8e5272422781fd65d6f72b2619, State: Initialized, Role: FOLLOWER
I20260620 14:49:25.704890 16375 raft_consensus.cc:348] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } }
I20260620 14:49:25.704949 16375 raft_consensus.cc:374] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:49:25.704986 16375 raft_consensus.cc:729] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce64d8a46f8f4dd0910203fc20d0646f, State: Initialized, Role: FOLLOWER
I20260620 14:49:25.704972 16646 consensus_queue.cc:260] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } }
I20260620 14:49:25.705026 16375 consensus_queue.cc:260] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } }
I20260620 14:49:25.705054 16646 ts_tablet_manager.cc:1428] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:25.705108 16375 ts_tablet_manager.cc:1428] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:49:25.707263 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:25.702150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34188 (local address 127.0.98.2:44095)
0620 14:49:25.704286 (+ 2136us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:25.704291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:25.704302 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:49:25.704341 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:25.704343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:25.704360 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:25.704439 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:25.704549 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.705590 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.706452 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.706937 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.706996 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:25.707054 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:25.707147 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 14:49:25.707183 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2099,"thread_start_us":95,"threads_started":1}
I20260620 14:49:25.707242 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:25.702054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44095 (local address 127.0.0.1:34188)
0620 14:49:25.703641 (+ 1587us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:25.703652 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:25.703690 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:25.704455 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:25.704456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:25.704458 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:25.704496 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:25.704497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.705649 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.705650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:25.706375 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:25.706377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:25.707008 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:25.707009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:25.707069 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:25.707088 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:49:25.707134 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 14:49:25.707175 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1572}
I20260620 14:49:25.707931 16555 tablet_service.cc:1467] Processing CreateTablet for tablet 06b424d310434fae80da1628fdf6d0b6 (DEFAULT_TABLE table=test-table [id=c38557672a0545678bcf48b6bf1cc161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:49:25.708215 16555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b424d310434fae80da1628fdf6d0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:49:25.710820 16766 tablet_bootstrap.cc:492] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d: Bootstrap starting.
I20260620 14:49:25.711360 16766 tablet_bootstrap.cc:654] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d: Neither blocks nor log segments found. Creating new log.
I20260620 14:49:25.711862 16766 log.cc:824] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:25.712631 16766 tablet_bootstrap.cc:492] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d: No bootstrap required, opened a new log
I20260620 14:49:25.712695 16766 ts_tablet_manager.cc:1397] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:49:25.714108 16766 raft_consensus.cc:348] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } }
I20260620 14:49:25.714262 16766 raft_consensus.cc:374] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:49:25.714283 16766 raft_consensus.cc:729] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d60b6c9420004fd488ec636183caea7d, State: Initialized, Role: FOLLOWER
I20260620 14:49:25.714390 16766 consensus_queue.cc:260] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } }
I20260620 14:49:25.714625 16766 ts_tablet_manager.cc:1428] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:49:25.714650 16621 heartbeater.cc:503] Master 127.0.98.62:40171 was elected leader, sending a full tablet report...
W20260620 14:49:25.807248 16622 tablet.cc:2369] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:49:25.926945 16484 tablet.cc:2369] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:49:25.928743 16756 tablet.cc:2369] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:49:26.109010 16488 raft_consensus.cc:482] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:49:26.109184 16488 raft_consensus.cc:504] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } }
I20260620 14:49:26.109534 16488 leader_election.cc:288] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d60b6c9420004fd488ec636183caea7d (127.0.98.2:44095), 57698b8e5272422781fd65d6f72b2619 (127.0.98.3:42949)
I20260620 14:49:26.112579 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:26.109836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44489 (local address 127.0.98.3:42949)
0620 14:49:26.109898 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:26.109899 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:26.109909 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:49:26.109981 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:26.109982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:26.110005 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:26.110072 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:26.110221 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:26.110754 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:26.111855 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:26.112274 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:26.112344 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:26.112422 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:26.112447 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 14:49:26.112488 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:49:26.112587 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:26.109762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43019 (local address 127.0.98.2:44095)
0620 14:49:26.109807 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:26.109809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:26.110022 (+ 213us) server_negotiation.cc:407] Connection header received
0620 14:49:26.110072 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:26.110073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:26.110086 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:26.110133 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:26.110263 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:26.110751 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:26.111878 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:26.112239 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:26.112295 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:26.112305 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:26.112479 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 14:49:26.112523 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:49:26.112588 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:26.109710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44095 (local address 127.0.98.1:43019)
0620 14:49:26.109947 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:26.109956 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:26.110019 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:26.110143 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:26.110143 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:26.110145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:26.110203 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:26.110206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:26.110832 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:26.110833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:26.111770 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:26.111772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:26.112314 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:26.112316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:26.112389 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:26.112411 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:49:26.112465 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:49:26.112506 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":42,"thread_start_us":67,"threads_started":1}
I20260620 14:49:26.112600 16769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:26.109708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42949 (local address 127.0.98.1:44489)
0620 14:49:26.109845 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:26.109856 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:26.109919 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:26.110090 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:26.110091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:26.110093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:26.110156 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:26.110159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:26.110842 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:26.110843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:26.111767 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:26.111769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:26.112358 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:26.112359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:26.112409 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:26.112413 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:49:26.112443 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 14:49:26.112488 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":10,"thread_start_us":60,"threads_started":1}
I20260620 14:49:26.112912 16575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b424d310434fae80da1628fdf6d0b6" candidate_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d60b6c9420004fd488ec636183caea7d" is_pre_election: true
I20260620 14:49:26.113102 16575 raft_consensus.cc:2454] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce64d8a46f8f4dd0910203fc20d0646f in term 0.
I20260620 14:49:26.113068 16709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b424d310434fae80da1628fdf6d0b6" candidate_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57698b8e5272422781fd65d6f72b2619" is_pre_election: true
I20260620 14:49:26.113189 16709 raft_consensus.cc:2454] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce64d8a46f8f4dd0910203fc20d0646f in term 0.
I20260620 14:49:26.113312 16372 leader_election.cc:302] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce64d8a46f8f4dd0910203fc20d0646f, d60b6c9420004fd488ec636183caea7d; no voters:
I20260620 14:49:26.113477 16488 raft_consensus.cc:2781] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:49:26.113530 16488 raft_consensus.cc:482] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:49:26.113552 16488 raft_consensus.cc:3037] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:49:26.114202 16488 raft_consensus.cc:504] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } }
I20260620 14:49:26.114332 16488 leader_election.cc:288] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 1 election: Requested vote from peers d60b6c9420004fd488ec636183caea7d (127.0.98.2:44095), 57698b8e5272422781fd65d6f72b2619 (127.0.98.3:42949)
I20260620 14:49:26.114485 16709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b424d310434fae80da1628fdf6d0b6" candidate_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57698b8e5272422781fd65d6f72b2619"
I20260620 14:49:26.114500 16575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b424d310434fae80da1628fdf6d0b6" candidate_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d60b6c9420004fd488ec636183caea7d"
I20260620 14:49:26.114570 16709 raft_consensus.cc:3037] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:49:26.114573 16575 raft_consensus.cc:3037] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:49:26.115296 16575 raft_consensus.cc:2454] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce64d8a46f8f4dd0910203fc20d0646f in term 1.
I20260620 14:49:26.115381 16709 raft_consensus.cc:2454] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce64d8a46f8f4dd0910203fc20d0646f in term 1.
I20260620 14:49:26.115552 16369 leader_election.cc:302] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57698b8e5272422781fd65d6f72b2619, ce64d8a46f8f4dd0910203fc20d0646f; no voters:
I20260620 14:49:26.115664 16488 raft_consensus.cc:2781] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:49:26.115795 16488 raft_consensus.cc:686] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 LEADER]: Becoming Leader. State: Replica: ce64d8a46f8f4dd0910203fc20d0646f, State: Running, Role: LEADER
I20260620 14:49:26.115927 16488 consensus_queue.cc:237] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } }
I20260620 14:49:26.116690 16294 catalog_manager.cc:5495] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f reported cstate change: term changed from 0 to 1, leader changed from <none> to ce64d8a46f8f4dd0910203fc20d0646f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } health_report { overall_health: UNKNOWN } } }
I20260620 14:49:26.145010 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:26.142598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50262 (local address 127.0.98.1:43327)
0620 14:49:26.142775 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:26.142777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:26.142783 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:49:26.142828 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:26.142828 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:26.142848 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:26.142926 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:49:26.143065 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:26.143606 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:26.144317 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:26.144521 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:26.144575 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:26.144722 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:49:26.144777 (+ 55us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 14:49:26.144915 (+ 138us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50262: token expired
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":99,"threads_started":1}
W20260620 14:49:26.145084 16773 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50262: token expired
I20260620 14:49:26.150780 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:26.145391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.0.98.62:40171)
0620 14:49:26.145446 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:26.145449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:26.145457 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:49:26.145499 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:26.145500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:26.145510 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:26.145557 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:26.146071 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:26.146535 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:26.147187 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:26.147373 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:26.147457 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:26.148638 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:26.148639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:26.148639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:26.148646 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:26.149784 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:26.150087 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:26.150088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:26.150088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:26.150139 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:26.150297 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:26.150298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:26.150298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:26.150404 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:26.150491 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:26.150608 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 14:49:26.150690 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:49:26.153687 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:26.151664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50274 (local address 127.0.98.1:43327)
0620 14:49:26.151703 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:26.151704 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:26.151713 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:49:26.151751 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:26.151752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:26.151761 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:26.151800 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:49:26.151901 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:26.152364 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:26.153034 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:26.153220 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:26.153271 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:26.153389 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:49:26.153495 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:49:26.153535 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:26.153590 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 14:49:26.153611 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:49:26.155494 16709 raft_consensus.cc:1261] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619 [term 1 FOLLOWER]: Refusing update from remote peer ce64d8a46f8f4dd0910203fc20d0646f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:49:26.155494 16575 raft_consensus.cc:1261] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d [term 1 FOLLOWER]: Refusing update from remote peer ce64d8a46f8f4dd0910203fc20d0646f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:49:26.155818 16771 consensus_queue.cc:1035] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:49:26.155999 16488 consensus_queue.cc:1035] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:49:26.158427 16775 mvcc.cc:205] Tried to move back new op lower bound from 7298936693369720832 to 7298936693211500544. Current Snapshot: MvccSnapshot[applied={T|T < 7298936693369720832}]
I20260620 14:49:26.160174 16294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:49:26.160261 16294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:49:26.160951 16294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: Sending DeleteTablet for 3 replicas of tablet 06b424d310434fae80da1628fdf6d0b6
I20260620 14:49:26.161244 16689 tablet_service.cc:1514] Processing DeleteTablet for tablet 06b424d310434fae80da1628fdf6d0b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:49:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50548
I20260620 14:49:26.161250 16402 tablet_service.cc:1514] Processing DeleteTablet for tablet 06b424d310434fae80da1628fdf6d0b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:49:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50248
I20260620 14:49:26.161329 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 16333
I20260620 14:49:26.161763 16762 tablet_replica.cc:330] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619: stopping tablet replica
I20260620 14:49:26.161818 16762 raft_consensus.cc:2229] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:49:26.161867 16762 raft_consensus.cc:2258] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:49:26.162027 16555 tablet_service.cc:1514] Processing DeleteTablet for tablet 06b424d310434fae80da1628fdf6d0b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:49:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34188
I20260620 14:49:26.162144 16762 ts_tablet_manager.cc:1905] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:49:26.162235 16785 tablet_replica.cc:330] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d: stopping tablet replica
I20260620 14:49:26.162328 16785 raft_consensus.cc:2229] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:49:26.162448 16785 raft_consensus.cc:2258] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:49:26.162850 16785 ts_tablet_manager.cc:1905] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:49:26.163712 16762 ts_tablet_manager.cc:1918] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:49:26.163770 16762 log.cc:1196] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/06b424d310434fae80da1628fdf6d0b6
I20260620 14:49:26.163986 16762 ts_tablet_manager.cc:1939] T 06b424d310434fae80da1628fdf6d0b6 P 57698b8e5272422781fd65d6f72b2619: Deleting consensus metadata
I20260620 14:49:26.164410 16785 ts_tablet_manager.cc:1918] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:49:26.164413 16279 catalog_manager.cc:4841] TS 57698b8e5272422781fd65d6f72b2619 (127.0.98.3:42949): tablet 06b424d310434fae80da1628fdf6d0b6 (table test-table [id=c38557672a0545678bcf48b6bf1cc161]) successfully deleted
I20260620 14:49:26.164485 16785 log.cc:1196] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/06b424d310434fae80da1628fdf6d0b6
I20260620 14:49:26.164695 16785 ts_tablet_manager.cc:1939] T 06b424d310434fae80da1628fdf6d0b6 P d60b6c9420004fd488ec636183caea7d: Deleting consensus metadata
I20260620 14:49:26.165055 16282 catalog_manager.cc:4841] TS d60b6c9420004fd488ec636183caea7d (127.0.98.2:44095): tablet 06b424d310434fae80da1628fdf6d0b6 (table test-table [id=c38557672a0545678bcf48b6bf1cc161]) successfully deleted
W20260620 14:49:26.167064 16280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43327 (error 108)
I20260620 14:49:26.167354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 16489
I20260620 14:49:26.167635 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:26.167519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43327 (local address 127.0.0.1:50286)
0620 14:49:26.167570 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:26.167606 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 14:49:26.167752 16280 catalog_manager.cc:4584] TS ce64d8a46f8f4dd0910203fc20d0646f (127.0.98.1:43327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 06b424d310434fae80da1628fdf6d0b6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43327: connect: Connection refused (error 111)
I20260620 14:49:26.172235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 16624
I20260620 14:49:26.176483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 16262
I20260620 14:49:46.181631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40171
--webserver_interface=127.0.98.62
--webserver_port=37505
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:49:46.259379 16787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:46.259594 16787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:46.259615 16787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:46.260695 16787 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:49:46.260727 16787 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:49:46.260740 16787 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:49:46.260749 16787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:49:46.260761 16787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:46.260782 16787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:46.260792 16787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:49:46.260803 16787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:49:46.262213 16787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:49:46.262399 16787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:46.262566 16787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:46.267278 16792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:46.267343 16795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:46.267320 16787 server_base.cc:1034] running on GCE node
W20260620 14:49:46.267457 16793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:46.267819 16787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:46.268119 16787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:46.269260 16787 hybrid_clock.cc:648] HybridClock initialized: now 1781966986269235 us; error 42 us; skew 500 ppm
Jun 20 14:49:46 dist-test-slave-cx89 krb5kdc[15717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:46.271620 16787 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:49:46.272018 16787 webserver.cc:460] Webserver started at http://127.0.98.62:37505/ using document root <none> and password file <none>
I20260620 14:49:46.272213 16787 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:46.272259 16787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:46.273561 16787 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:49:46.274242 16802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:46.274430 16787 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:49:46.274528 16787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "080535b714844b028b54559fceca82a0"
format_stamp: "Formatted at 2026-06-20 14:49:04 on dist-test-slave-cx89"
server_key: "45ece8f915c105a9d37fa7d7fa836567"
server_key_iv: "d8d04417ab01d4e6cbb15a0287bd8c51"
server_key_version: "encryptionkey@0"
I20260620 14:49:46.274868 16787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:46.284649 16787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:46.286620 16787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:46.286749 16787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:46.290446 16787 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40171
I20260620 14:49:46.290481 16854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40171 every 8 connection(s)
I20260620 14:49:46.290822 16787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:49:46.292244 16855 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:49:46.292816 16855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: Bootstrap starting.
I20260620 14:49:46.296131 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 16787
I20260620 14:49:46.296383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43327
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40171
--webserver_port=43173
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:49:46.301086 16855 log.cc:824] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:46.303201 16855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:49:46.303505 16855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: Bootstrap complete.
I20260620 14:49:46.305506 16855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080535b714844b028b54559fceca82a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40171 } }
I20260620 14:49:46.305728 16855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 080535b714844b028b54559fceca82a0, State: Initialized, Role: FOLLOWER
I20260620 14:49:46.305891 16855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080535b714844b028b54559fceca82a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40171 } }
I20260620 14:49:46.305966 16855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:49:46.306011 16855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:49:46.306056 16855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:49:46.306994 16855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080535b714844b028b54559fceca82a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40171 } }
I20260620 14:49:46.307085 16855 leader_election.cc:302] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 080535b714844b028b54559fceca82a0; no voters:
I20260620 14:49:46.307245 16855 leader_election.cc:288] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:49:46.307296 16859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:49:46.307479 16859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [term 3 LEADER]: Becoming Leader. State: Replica: 080535b714844b028b54559fceca82a0, State: Running, Role: LEADER
I20260620 14:49:46.307525 16855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:49:46.307576 16859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080535b714844b028b54559fceca82a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40171 } }
I20260620 14:49:46.307837 16859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 080535b714844b028b54559fceca82a0. Latest consensus state: current_term: 3 leader_uuid: "080535b714844b028b54559fceca82a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080535b714844b028b54559fceca82a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40171 } } }
I20260620 14:49:46.307906 16859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:49:46.308034 16860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "080535b714844b028b54559fceca82a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080535b714844b028b54559fceca82a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40171 } } }
I20260620 14:49:46.308097 16860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:49:46.309198 16875 catalog_manager.cc:1259] Loaded cluster ID: 3326b935460342819d44e6a40743ed1f
I20260620 14:49:46.309243 16875 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: loading cluster ID for follower catalog manager: success
I20260620 14:49:46.310362 16875 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 080535b714844b028b54559fceca82a0: acquiring CA information for follower catalog manager: success
F20260620 14:49:46.310750 16875 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe3c2f809ad google::LogMessage::Fail() at ??:0
@ 0x7fe3c2f8388a google::LogMessage::SendToLog() at ??:0
@ 0x7fe3c2f804de google::LogMessage::Flush() at ??:0
@ 0x7fe3c2f81a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe3c56ed104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe3caed96b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fe3caed9989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fe3caed9e4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fe3caedc8c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fe3c4e43954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe3c5ca76db start_thread at ??:0
@ 0x7fe3c1a7e61f clone at ??:0
W20260620 14:49:46.382921 16857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:46.383114 16857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:46.383145 16857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:46.384308 16857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:46.384352 16857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:46.384387 16857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:49:46.385816 16857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43173
--tserver_master_addrs=127.0.98.62:40171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:49:46.386047 16857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:46.386247 16857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:46.391135 16896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:46.391166 16893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:46.391211 16857 server_base.cc:1034] running on GCE node
W20260620 14:49:46.391407 16894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:46.391606 16857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:46.391842 16857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:46.393013 16857 hybrid_clock.cc:648] HybridClock initialized: now 1781966986392996 us; error 31 us; skew 500 ppm
Jun 20 14:49:46 dist-test-slave-cx89 krb5kdc[15717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:46.395339 16857 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:49:46.395803 16857 webserver.cc:460] Webserver started at http://127.0.98.1:43173/ using document root <none> and password file <none>
I20260620 14:49:46.396030 16857 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:46.396071 16857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:46.397298 16857 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 14:49:46.397893 16903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:46.398070 16857 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:49:46.398171 16857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "ce64d8a46f8f4dd0910203fc20d0646f"
format_stamp: "Formatted at 2026-06-20 14:49:04 on dist-test-slave-cx89"
server_key: "3d13e8c51f583dab4c4bad25c01dfbfc"
server_key_iv: "0b43c15d5d092d0abecf103a0750949c"
server_key_version: "encryptionkey@0"
I20260620 14:49:46.398609 16857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:46.434108 16857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:46.436591 16857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:46.436757 16857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:46.437019 16857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:49:46.437525 16910 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:49:46.438915 16857 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:49:46.438972 16857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:49:46.439005 16857 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:49:46.439869 16857 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:49:46.439913 16857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260620 14:49:46.439966 16910 tablet_bootstrap.cc:492] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f: Bootstrap starting.
I20260620 14:49:46.446800 16857 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43327
I20260620 14:49:46.447014 17017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43327 every 8 connection(s)
I20260620 14:49:46.447291 16857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:49:46.449796 16910 log.cc:824] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:46.451027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 16857
I20260620 14:49:46.451304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44095
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40171
--webserver_port=45453
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:49:46.452139 16910 tablet_bootstrap.cc:492] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:49:46.452400 16910 tablet_bootstrap.cc:492] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f: Bootstrap complete.
I20260620 14:49:46.453464 16910 ts_tablet_manager.cc:1397] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260620 14:49:46.454509 16910 raft_consensus.cc:348] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } }
I20260620 14:49:46.454731 16910 raft_consensus.cc:729] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce64d8a46f8f4dd0910203fc20d0646f, State: Initialized, Role: FOLLOWER
I20260620 14:49:46.454869 16910 consensus_queue.cc:260] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } }
I20260620 14:49:46.455122 16910 ts_tablet_manager.cc:1428] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260620 14:49:46.515224 17020 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:49:46.447936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:34903)
0620 14:49:46.448098 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:46.448144 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:46.448776 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:46.515020 (+ 66244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
W20260620 14:49:46.515484 17018 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40171 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40171: Client connection negotiation failed: client connection to 127.0.98.62:40171: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:49:46.515810 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:46.515704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:40083)
0620 14:49:46.515750 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:46.515788 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 14:49:46.516086 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:46.516006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:43603)
0620 14:49:46.516043 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:46.516065 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 14:49:46.516188 17018 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:49:46.545931 17022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:46.546150 17022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:46.546185 17022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:46.547384 17022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:46.547444 17022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:46.547484 17022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:49:46.548985 17022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45453
--tserver_master_addrs=127.0.98.62:40171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:49:46.549233 17022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:46.549443 17022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:46.554184 17028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:46.554245 17031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:46.554296 17022 server_base.cc:1034] running on GCE node
W20260620 14:49:46.554250 17029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:46.554580 17022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:46.554814 17022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:46.555948 17022 hybrid_clock.cc:648] HybridClock initialized: now 1781966986555929 us; error 30 us; skew 500 ppm
Jun 20 14:49:46 dist-test-slave-cx89 krb5kdc[15717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:46.558271 17022 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:49:46.558712 17022 webserver.cc:460] Webserver started at http://127.0.98.2:45453/ using document root <none> and password file <none>
I20260620 14:49:46.558914 17022 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:46.558961 17022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:46.560259 17022 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:49:46.560858 17038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:46.561025 17022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:49:46.561103 17022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "d60b6c9420004fd488ec636183caea7d"
format_stamp: "Formatted at 2026-06-20 14:49:04 on dist-test-slave-cx89"
server_key: "40a102b1de1284eaed21a42280eecc7d"
server_key_iv: "c756c9e3c1bc91da32ce119fcdff8f07"
server_key_version: "encryptionkey@0"
I20260620 14:49:46.561429 17022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:46.572263 17022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:46.573976 17022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:46.574105 17022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:46.574304 17022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:49:46.574574 17022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:49:46.574615 17022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:46.574645 17022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:49:46.574668 17022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:46.580502 17022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44095
I20260620 14:49:46.580566 17151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44095 every 8 connection(s)
I20260620 14:49:46.580907 17022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:49:46.581959 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:46.581574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:60405)
0620 14:49:46.581743 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:46.581815 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
W20260620 14:49:46.582345 17152 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40171 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40171: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
I20260620 14:49:46.582677 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:46.582552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:43649)
0620 14:49:46.582616 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:46.582647 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 14:49:46.582943 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:46.582886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:55949)
0620 14:49:46.582908 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:46.582930 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260620 14:49:46.583030 17152 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:49:46.588263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 17022
I20260620 14:49:46.588521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42949
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40171
--webserver_port=42277
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:49:46.666677 17155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:46.666877 17155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:46.666898 17155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:46.668015 17155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:46.668051 17155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:46.668083 17155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:49:46.669472 17155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42277
--tserver_master_addrs=127.0.98.62:40171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:49:46.669656 17155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:46.669831 17155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:46.674566 17161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:46.674561 17163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:46.674566 17160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:46.674666 17155 server_base.cc:1034] running on GCE node
I20260620 14:49:46.674965 17155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:46.675173 17155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:46.676318 17155 hybrid_clock.cc:648] HybridClock initialized: now 1781966986676304 us; error 28 us; skew 500 ppm
Jun 20 14:49:46 dist-test-slave-cx89 krb5kdc[15717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:46.678678 17155 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:49:46.679143 17155 webserver.cc:460] Webserver started at http://127.0.98.3:42277/ using document root <none> and password file <none>
I20260620 14:49:46.679320 17155 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:46.679359 17155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:46.680683 17155 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:49:46.681326 17170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:46.681545 17155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:49:46.681645 17155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "57698b8e5272422781fd65d6f72b2619"
format_stamp: "Formatted at 2026-06-20 14:49:05 on dist-test-slave-cx89"
server_key: "82269baba8eeed589205155af8d5dfea"
server_key_iv: "58640e68c623ffd2da51e2321986c11a"
server_key_version: "encryptionkey@0"
I20260620 14:49:46.681995 17155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:46.699221 17155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:46.701143 17155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:46.701280 17155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:46.701498 17155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:49:46.701804 17155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:49:46.701836 17155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:46.701865 17155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:49:46.701882 17155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:46.707370 17155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42949
I20260620 14:49:46.707468 17283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42949 every 8 connection(s)
I20260620 14:49:46.707775 17155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:49:46.708750 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:46.708394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:59173)
0620 14:49:46.708539 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:46.708612 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
W20260620 14:49:46.709131 17284 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40171 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40171: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
I20260620 14:49:46.709384 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:46.709290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:47919)
0620 14:49:46.709342 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:46.709367 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 14:49:46.709581 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:46.709532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:46965)
0620 14:49:46.709552 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:46.709569 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260620 14:49:46.709662 17284 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:49:46.713434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 17155
I20260620 14:49:47.516774 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:47.516533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:53085)
0620 14:49:47.516667 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:47.516707 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260620 14:49:47.583575 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:47.583336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:48259)
0620 14:49:47.583479 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:47.583524 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":53,"threads_started":1}
I20260620 14:49:47.710186 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:47.709988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:51095)
0620 14:49:47.710099 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:47.710138 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
I20260620 14:49:47.803030 17291 raft_consensus.cc:482] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:49:47.803146 17291 raft_consensus.cc:504] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } }
I20260620 14:49:47.803440 17291 leader_election.cc:288] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d60b6c9420004fd488ec636183caea7d (127.0.98.2:44095), 57698b8e5272422781fd65d6f72b2619 (127.0.98.3:42949)
Jun 20 14:49:47 dist-test-slave-cx89 krb5kdc[15717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 20 14:49:47 dist-test-slave-cx89 krb5kdc[15717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 14:49:47.813472 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:47.803795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60567 (local address 127.0.98.2:44095)
0620 14:49:47.803937 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:47.803946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:47.803972 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:49:47.804404 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:47.804405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:47.804446 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:47.804567 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:47.805330 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:47.806175 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:47.807157 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:47.807605 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:47.808102 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:47.810476 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:47.810480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:47.810480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:47.810492 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:47.812062 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:47.812341 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:47.812343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:47.812344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:47.812417 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:47.812675 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:47.812676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:47.812676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:47.812893 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:47.812961 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:47.813245 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 14:49:47.813354 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260620 14:49:47.813463 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:47.803616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44095 (local address 127.0.98.1:60567)
0620 14:49:47.803678 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:47.803689 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:47.803744 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:47.804635 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:47.804638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:47.805011 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:49:47.805215 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:47.805221 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:47.806277 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:47.806278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:47.807065 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:47.807067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:47.807695 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:47.807696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:47.807831 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:47.808389 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:49:47.808398 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:49:47.810362 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:49:47.812139 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:47.812140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:47.812140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:47.812284 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:47.812485 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:47.812486 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:47.812486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:47.812614 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:47.812968 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:49:47.812974 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:49:47.813104 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 14:49:47.813181 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:49:47.813365 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":79}
I20260620 14:49:47.813483 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:47.803751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40623 (local address 127.0.98.3:42949)
0620 14:49:47.803897 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:47.803929 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:47.803967 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:49:47.804404 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:47.804406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:47.804446 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:47.804561 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:47.805342 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:47.806174 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:47.807157 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:47.807599 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:47.807712 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:47.810116 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:47.810120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:47.810121 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:47.810132 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:47.811776 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:47.812358 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:47.812360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:47.812360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:47.812437 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:47.812682 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:47.812683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:47.812684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:47.812899 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:47.812967 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:47.813250 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 14:49:47.813364 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260620 14:49:47.813563 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:47.803628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42949 (local address 127.0.98.1:40623)
0620 14:49:47.803765 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:47.803777 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:47.803844 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:47.804662 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:47.804663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:47.805017 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:49:47.805216 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:47.805223 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:47.806287 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:47.806288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:47.807064 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:47.807066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:47.807684 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:47.807686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:47.807831 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:47.808451 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:49:47.808454 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:49:47.810005 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:49:47.811853 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:47.811859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:47.811861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:47.812068 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:47.812507 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:47.812507 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:47.812508 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:47.812618 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:47.812968 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:49:47.812975 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:49:47.813095 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 14:49:47.813180 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:49:47.813438 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":139,"thread_start_us":73,"threads_started":1}
W20260620 14:49:47.814742 16907 leader_election.cc:341] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d60b6c9420004fd488ec636183caea7d (127.0.98.2:44095): Not found: Tablet not found: 06b424d310434fae80da1628fdf6d0b6
W20260620 14:49:47.814890 16904 leader_election.cc:341] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 57698b8e5272422781fd65d6f72b2619 (127.0.98.3:42949): Not found: Tablet not found: 06b424d310434fae80da1628fdf6d0b6
I20260620 14:49:47.814937 16904 leader_election.cc:302] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce64d8a46f8f4dd0910203fc20d0646f; no voters: 57698b8e5272422781fd65d6f72b2619, d60b6c9420004fd488ec636183caea7d
I20260620 14:49:47.815048 17291 raft_consensus.cc:2726] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:49:48.517421 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:48.517216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:54675)
0620 14:49:48.517337 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:48.517376 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
I20260620 14:49:48.584225 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:48.584002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:57101)
0620 14:49:48.584147 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:48.584184 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260620 14:49:48.710819 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:48.710613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:46787)
0620 14:49:48.710746 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:48.710780 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260620 14:49:49.505167 17298 raft_consensus.cc:482] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:49:49.505260 17298 raft_consensus.cc:504] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } }
I20260620 14:49:49.505424 17298 leader_election.cc:288] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d60b6c9420004fd488ec636183caea7d (127.0.98.2:44095), 57698b8e5272422781fd65d6f72b2619 (127.0.98.3:42949)
W20260620 14:49:49.505848 16907 leader_election.cc:341] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d60b6c9420004fd488ec636183caea7d (127.0.98.2:44095): Not found: Tablet not found: 06b424d310434fae80da1628fdf6d0b6
W20260620 14:49:49.505937 16904 leader_election.cc:341] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 57698b8e5272422781fd65d6f72b2619 (127.0.98.3:42949): Not found: Tablet not found: 06b424d310434fae80da1628fdf6d0b6
I20260620 14:49:49.505982 16904 leader_election.cc:302] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce64d8a46f8f4dd0910203fc20d0646f; no voters: 57698b8e5272422781fd65d6f72b2619, d60b6c9420004fd488ec636183caea7d
I20260620 14:49:49.506049 17298 raft_consensus.cc:2726] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:49:49.518052 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:49.517849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:54733)
0620 14:49:49.517974 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:49.518008 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260620 14:49:49.584914 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:49.584662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:39513)
0620 14:49:49.584830 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:49.584871 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260620 14:49:49.711475 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:49.711231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:58631)
0620 14:49:49.711371 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:49.711408 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":58,"threads_started":1}
I20260620 14:49:50.518766 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:50.518526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:45137)
0620 14:49:50.518672 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:50.518720 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
I20260620 14:49:50.585638 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:50.585384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:59113)
0620 14:49:50.585542 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:50.585588 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260620 14:49:50.712160 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:50.711917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:43171)
0620 14:49:50.712081 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:50.712121 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":75,"threads_started":1}
I20260620 14:49:51.519517 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:51.519264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:55699)
0620 14:49:51.519402 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:51.519466 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260620 14:49:51.586292 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:51.586088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:42757)
0620 14:49:51.586209 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:51.586246 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260620 14:49:51.712939 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:51.712693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:48591)
0620 14:49:51.712852 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:51.712894 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":65,"threads_started":1}
I20260620 14:49:52.321497 17308 raft_consensus.cc:482] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:49:52.321596 17308 raft_consensus.cc:504] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } }
I20260620 14:49:52.321775 17308 leader_election.cc:288] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d60b6c9420004fd488ec636183caea7d (127.0.98.2:44095), 57698b8e5272422781fd65d6f72b2619 (127.0.98.3:42949)
W20260620 14:49:52.322140 16907 leader_election.cc:341] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d60b6c9420004fd488ec636183caea7d (127.0.98.2:44095): Not found: Tablet not found: 06b424d310434fae80da1628fdf6d0b6
W20260620 14:49:52.322218 16904 leader_election.cc:341] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 57698b8e5272422781fd65d6f72b2619 (127.0.98.3:42949): Not found: Tablet not found: 06b424d310434fae80da1628fdf6d0b6
I20260620 14:49:52.322286 16904 leader_election.cc:302] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce64d8a46f8f4dd0910203fc20d0646f; no voters: 57698b8e5272422781fd65d6f72b2619, d60b6c9420004fd488ec636183caea7d
I20260620 14:49:52.322365 17308 raft_consensus.cc:2726] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:49:52.520251 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:52.520001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:41747)
0620 14:49:52.520154 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:52.520201 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260620 14:49:52.587028 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:52.586809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:55947)
0620 14:49:52.586953 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:52.586986 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260620 14:49:52.713599 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:52.713382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:52441)
0620 14:49:52.713520 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:52.713555 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260620 14:49:53.520953 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:53.520756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:44145)
0620 14:49:53.520867 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:53.520905 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
I20260620 14:49:53.587705 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:53.587490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:39127)
0620 14:49:53.587619 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:53.587658 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
I20260620 14:49:53.714295 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:53.714072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:42051)
0620 14:49:53.714188 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:53.714251 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260620 14:49:54.521579 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:54.521386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:46821)
0620 14:49:54.521498 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:54.521531 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260620 14:49:54.588316 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:54.588117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:59643)
0620 14:49:54.588227 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:54.588263 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260620 14:49:54.714954 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:54.714714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:40815)
0620 14:49:54.714865 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:54.714908 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260620 14:49:55.522279 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:55.522036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:48621)
0620 14:49:55.522189 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:55.522232 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":75,"threads_started":1}
I20260620 14:49:55.588919 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:55.588716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:38921)
0620 14:49:55.588842 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:55.588876 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260620 14:49:55.715675 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:55.715487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:55083)
0620 14:49:55.715600 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:55.715634 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":54,"threads_started":1}
I20260620 14:49:56.237828 17321 raft_consensus.cc:482] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:49:56.237917 17321 raft_consensus.cc:504] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60b6c9420004fd488ec636183caea7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44095 } } peers { permanent_uuid: "ce64d8a46f8f4dd0910203fc20d0646f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43327 } } peers { permanent_uuid: "57698b8e5272422781fd65d6f72b2619" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42949 } }
I20260620 14:49:56.238044 17321 leader_election.cc:288] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d60b6c9420004fd488ec636183caea7d (127.0.98.2:44095), 57698b8e5272422781fd65d6f72b2619 (127.0.98.3:42949)
W20260620 14:49:56.238445 16907 leader_election.cc:341] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d60b6c9420004fd488ec636183caea7d (127.0.98.2:44095): Not found: Tablet not found: 06b424d310434fae80da1628fdf6d0b6
W20260620 14:49:56.238533 16904 leader_election.cc:341] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 57698b8e5272422781fd65d6f72b2619 (127.0.98.3:42949): Not found: Tablet not found: 06b424d310434fae80da1628fdf6d0b6
I20260620 14:49:56.238580 16904 leader_election.cc:302] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce64d8a46f8f4dd0910203fc20d0646f; no voters: 57698b8e5272422781fd65d6f72b2619, d60b6c9420004fd488ec636183caea7d
I20260620 14:49:56.238655 17321 raft_consensus.cc:2726] T 06b424d310434fae80da1628fdf6d0b6 P ce64d8a46f8f4dd0910203fc20d0646f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:49:56.522957 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:56.522709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:40453)
0620 14:49:56.522870 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:56.522911 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":64,"threads_started":1}
I20260620 14:49:56.589648 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:56.589416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:36161)
0620 14:49:56.589546 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:56.589590 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260620 14:49:56.716368 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:56.716125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:36375)
0620 14:49:56.716269 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:56.716312 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":59,"threads_started":1}
I20260620 14:49:57.523720 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:57.523513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:50129)
0620 14:49:57.523636 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:57.523674 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260620 14:49:57.590283 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:57.590082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:33199)
0620 14:49:57.590202 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:57.590236 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260620 14:49:57.717036 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:57.716820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:57843)
0620 14:49:57.716954 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:57.716991 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260620 14:49:58.524430 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:58.524195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:50385)
0620 14:49:58.524328 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:58.524372 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260620 14:49:58.590991 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:58.590752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:37479)
0620 14:49:58.590902 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:58.590944 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260620 14:49:58.717707 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:58.717485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:46985)
0620 14:49:58.717613 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:58.717659 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260620 14:49:59.525154 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:59.524912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:43723)
0620 14:49:59.525066 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:59.525107 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
I20260620 14:49:59.591674 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:59.591449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:43837)
0620 14:49:59.591588 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:59.591628 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260620 14:49:59.718407 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:59.718194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:36687)
0620 14:49:59.718328 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:59.718362 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260620 14:50:00.525840 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:00.525631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:56131)
0620 14:50:00.525758 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:00.525793 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260620 14:50:00.592358 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:00.592136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:36723)
0620 14:50:00.592273 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:00.592312 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":59,"threads_started":1}
I20260620 14:50:00.719206 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:00.718940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:56893)
0620 14:50:00.719109 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:00.719153 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
I20260620 14:50:01.526600 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:01.526365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.1:35083)
0620 14:50:01.526512 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:01.526555 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":57,"threads_started":1}
I20260620 14:50:01.593175 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:01.592934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.2:48717)
0620 14:50:01.593083 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:01.593127 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260620 14:50:01.719972 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:01.719734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40171 (local address 127.0.98.3:35369)
0620 14:50:01.719873 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:01.719927 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":59,"threads_started":1}
I20260620 14:50:01.720326 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 16857
I20260620 14:50:01.724356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 17022
I20260620 14:50:01.728001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 17155
2026-06-20T14:50:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58333 ms)
[----------] 30 tests from SecurityITest (253286 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:01 dist-test-slave-cx89 krb5kdc[17342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:01 dist-test-slave-cx89 krb5kdc[17342](info): set up 2 sockets
Jun 20 14:50:01 dist-test-slave-cx89 krb5kdc[17342](info): commencing operation
krb5kdc: starting...
W20260620 14:50:02.690828 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:02 dist-test-slave-cx89 krb5kdc[17342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:50:02.802135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40297
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:02.880903 17355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:02.881102 17355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:02.881124 17355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:02.882267 17355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:02.882308 17355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:02.882320 17355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:02.882333 17355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:02.882356 17355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:02.883862 17355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:02.884078 17355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:02.884279 17355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:02.888804 17363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:02.888867 17355 server_base.cc:1034] running on GCE node
W20260620 14:50:02.888804 17360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:02.888800 17361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:02.889329 17355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:02.889595 17355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:02.890758 17355 hybrid_clock.cc:648] HybridClock initialized: now 1781967002890747 us; error 29 us; skew 500 ppm
Jun 20 14:50:02 dist-test-slave-cx89 krb5kdc[17342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:02.893076 17355 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:50:02.893491 17355 webserver.cc:460] Webserver started at http://127.0.98.62:45905/ using document root <none> and password file <none>
I20260620 14:50:02.893703 17355 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:02.893750 17355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:02.893872 17355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:02.894882 17355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "23fe64bc691e4d9ca57979d0e10cd3cd"
format_stamp: "Formatted at 2026-06-20 14:50:02 on dist-test-slave-cx89"
server_key: "06b31a03a2850fae1fd9f515c6d88fa7"
server_key_iv: "248039b30d18f7cab9f1f4eaa7898e6a"
server_key_version: "encryptionkey@0"
I20260620 14:50:02.895198 17355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "23fe64bc691e4d9ca57979d0e10cd3cd"
format_stamp: "Formatted at 2026-06-20 14:50:02 on dist-test-slave-cx89"
server_key: "06b31a03a2850fae1fd9f515c6d88fa7"
server_key_iv: "248039b30d18f7cab9f1f4eaa7898e6a"
server_key_version: "encryptionkey@0"
I20260620 14:50:02.896493 17355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:50:02.897225 17370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:02.897377 17355 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:02.897455 17355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "23fe64bc691e4d9ca57979d0e10cd3cd"
format_stamp: "Formatted at 2026-06-20 14:50:02 on dist-test-slave-cx89"
server_key: "06b31a03a2850fae1fd9f515c6d88fa7"
server_key_iv: "248039b30d18f7cab9f1f4eaa7898e6a"
server_key_version: "encryptionkey@0"
I20260620 14:50:02.897527 17355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:02.931586 17355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:02.933421 17355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:02.933552 17355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:02.937203 17355 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40297
I20260620 14:50:02.937251 17422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40297 every 8 connection(s)
I20260620 14:50:02.937563 17355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:02.938411 17423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:02.940651 17423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd: Bootstrap starting.
I20260620 14:50:02.941187 17423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:02.941437 17423 log.cc:824] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:02.942087 17423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd: No bootstrap required, opened a new log
I20260620 14:50:02.943712 17423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23fe64bc691e4d9ca57979d0e10cd3cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40297 } }
I20260620 14:50:02.943848 17423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:02.943869 17423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23fe64bc691e4d9ca57979d0e10cd3cd, State: Initialized, Role: FOLLOWER
I20260620 14:50:02.944010 17423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23fe64bc691e4d9ca57979d0e10cd3cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40297 } }
I20260620 14:50:02.944078 17423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:02.944101 17423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:02.944146 17423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:02.944626 17423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23fe64bc691e4d9ca57979d0e10cd3cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40297 } }
I20260620 14:50:02.944718 17423 leader_election.cc:302] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23fe64bc691e4d9ca57979d0e10cd3cd; no voters:
I20260620 14:50:02.944865 17423 leader_election.cc:288] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:02.944900 17426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:02.945036 17426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd [term 1 LEADER]: Becoming Leader. State: Replica: 23fe64bc691e4d9ca57979d0e10cd3cd, State: Running, Role: LEADER
I20260620 14:50:02.945119 17423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:02.945127 17426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23fe64bc691e4d9ca57979d0e10cd3cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40297 } }
I20260620 14:50:02.945559 17428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23fe64bc691e4d9ca57979d0e10cd3cd. Latest consensus state: current_term: 1 leader_uuid: "23fe64bc691e4d9ca57979d0e10cd3cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23fe64bc691e4d9ca57979d0e10cd3cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40297 } } }
I20260620 14:50:02.945643 17428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:02.945689 17427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23fe64bc691e4d9ca57979d0e10cd3cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23fe64bc691e4d9ca57979d0e10cd3cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40297 } } }
I20260620 14:50:02.945762 17427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:02.945878 17435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:02.946369 17435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:02.947149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 17355
I20260620 14:50:02.947248 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:02.947464 392 external_mini_cluster.cc:1428] Setting key 2c99302988af258435f3df3fecf2a58d
I20260620 14:50:02.947947 17435 catalog_manager.cc:1347] Generated new cluster ID: 039a18f1783c4015b83ffdfbdd17be78
I20260620 14:50:02.947992 17435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:02.955322 17435 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:02.956012 17435 catalog_manager.cc:1504] Loading token signing keys...
Jun 20 14:50:02 dist-test-slave-cx89 krb5kdc[17342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:50:02.960781 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:02.947906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36912 (local address 127.0.98.62:40297)
0620 14:50:02.948042 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:02.948052 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:02.951928 (+ 3876us) server_negotiation.cc:407] Connection header received
0620 14:50:02.952050 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:02.952055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:02.952098 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:02.952212 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:02.952807 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:02.954348 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:02.955114 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:02.955676 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:02.955773 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:02.957885 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:02.957889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:02.957889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:02.957899 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:02.959333 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:02.959685 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:02.959688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:02.959688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:02.959803 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:02.959983 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:02.959984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:02.959985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:02.960196 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:02.960317 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:02.960423 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 14:50:02.960581 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":237,"server-negotiator.queue_time_us":83,"thread_start_us":62,"threads_started":1}
I20260620 14:50:02.965349 17435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd: Generated new TSK 0
I20260620 14:50:02.965549 17435 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:50:02.992024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40297
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:03.092348 17449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:03.092551 17449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:03.092583 17449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:03.093751 17449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:03.093787 17449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:03.093822 17449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:50:03.095501 17449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:03.095744 17449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:03.095955 17449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:03.100656 17457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:03.100677 17454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:03.100660 17455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:03.101058 17449 server_base.cc:1034] running on GCE node
I20260620 14:50:03.101230 17449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:03.101418 17449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:03.102566 17449 hybrid_clock.cc:648] HybridClock initialized: now 1781967003102556 us; error 29 us; skew 500 ppm
Jun 20 14:50:03 dist-test-slave-cx89 krb5kdc[17342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:03.104890 17449 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:50:03.105301 17449 webserver.cc:460] Webserver started at http://127.0.98.1:38629/ using document root <none> and password file <none>
I20260620 14:50:03.105489 17449 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:03.105531 17449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:03.105639 17449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:03.106642 17449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "511cb13ae334492bbcda9f6f78b5b633"
format_stamp: "Formatted at 2026-06-20 14:50:03 on dist-test-slave-cx89"
server_key: "fa961af1a27bd2c53e538f9f1fe3be7f"
server_key_iv: "293522c57073a097698db078b5b9c5b6"
server_key_version: "encryptionkey@0"
I20260620 14:50:03.106973 17449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "511cb13ae334492bbcda9f6f78b5b633"
format_stamp: "Formatted at 2026-06-20 14:50:03 on dist-test-slave-cx89"
server_key: "fa961af1a27bd2c53e538f9f1fe3be7f"
server_key_iv: "293522c57073a097698db078b5b9c5b6"
server_key_version: "encryptionkey@0"
I20260620 14:50:03.108211 17449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 14:50:03.108819 17464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:03.108963 17449 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:03.109040 17449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "511cb13ae334492bbcda9f6f78b5b633"
format_stamp: "Formatted at 2026-06-20 14:50:03 on dist-test-slave-cx89"
server_key: "fa961af1a27bd2c53e538f9f1fe3be7f"
server_key_iv: "293522c57073a097698db078b5b9c5b6"
server_key_version: "encryptionkey@0"
I20260620 14:50:03.109095 17449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:03.132673 17449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:03.134480 17449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:03.134615 17449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:03.134840 17449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:03.135126 17449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:03.135159 17449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:03.135187 17449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:03.135210 17449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:03.140561 17449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41337
I20260620 14:50:03.140617 17577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41337 every 8 connection(s)
I20260620 14:50:03.140899 17449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:50:03 dist-test-slave-cx89 krb5kdc[17342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:50:03.147307 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 17449
I20260620 14:50:03.147403 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:50:03.147658 392 external_mini_cluster.cc:1428] Setting key d0bc30db8851f8ef1479a5b535c99455
I20260620 14:50:03.150045 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.141578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44973 (local address 127.0.98.62:40297)
0620 14:50:03.141641 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:03.141643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:03.141836 (+ 193us) server_negotiation.cc:407] Connection header received
0620 14:50:03.142773 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:03.142775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:03.142788 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:03.142839 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:03.143479 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.143990 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.144960 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.145163 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.145218 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.147240 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:03.147242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:03.147242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:03.147250 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:03.148761 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:03.149134 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:03.149135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:03.149135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:03.149194 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:03.149410 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:03.149411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:03.149411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:03.149535 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:03.149591 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:03.149853 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 14:50:03.149954 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:50:03.150200 17580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.141542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40297 (local address 127.0.98.1:44973)
0620 14:50:03.141722 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:03.141758 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:03.142661 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:03.142927 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:03.142931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:03.143183 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:03.143372 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:03.143376 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.144070 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.144071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:03.144881 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:03.144884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.145230 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.145231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:03.145352 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.145758 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:03.145766 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:03.147139 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:03.148838 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:03.148845 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:03.148848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:03.149068 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:03.149247 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:03.149248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:03.149248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:03.149354 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:03.149590 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:03.149597 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:03.149735 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 14:50:03.149822 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:50:03.149975 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
I20260620 14:50:03.150643 17578 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40297
I20260620 14:50:03.150744 17578 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:03.150926 17578 heartbeater.cc:511] Master 127.0.98.62:40297 requested a full tablet report, sending...
I20260620 14:50:03.151590 17387 ts_manager.cc:194] Registered new tserver with Master: 511cb13ae334492bbcda9f6f78b5b633 (127.0.98.1:41337)
I20260620 14:50:03.152513 17387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44973
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:50:03.171399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40297
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:03.248935 17583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:03.249138 17583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:03.249161 17583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:03.250303 17583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:03.250344 17583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:03.250368 17583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:50:03.251857 17583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:03.252075 17583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:03.252279 17583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:03.256995 17591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:03.256966 17589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:03.257126 17583 server_base.cc:1034] running on GCE node
W20260620 14:50:03.256984 17588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:03.257393 17583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:03.257635 17583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:03.258797 17583 hybrid_clock.cc:648] HybridClock initialized: now 1781967003258779 us; error 29 us; skew 500 ppm
Jun 20 14:50:03 dist-test-slave-cx89 krb5kdc[17342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:03.261246 17583 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:50:03.261648 17583 webserver.cc:460] Webserver started at http://127.0.98.2:41221/ using document root <none> and password file <none>
I20260620 14:50:03.261837 17583 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:03.261883 17583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:03.262001 17583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:03.263007 17583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "41685603f78341fdb31565b6ed33e296"
format_stamp: "Formatted at 2026-06-20 14:50:03 on dist-test-slave-cx89"
server_key: "72b43c117789b92d5e0beec780e71fb4"
server_key_iv: "290fa3c2710009f4654eff4acc7fb2e5"
server_key_version: "encryptionkey@0"
I20260620 14:50:03.263338 17583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "41685603f78341fdb31565b6ed33e296"
format_stamp: "Formatted at 2026-06-20 14:50:03 on dist-test-slave-cx89"
server_key: "72b43c117789b92d5e0beec780e71fb4"
server_key_iv: "290fa3c2710009f4654eff4acc7fb2e5"
server_key_version: "encryptionkey@0"
I20260620 14:50:03.264575 17583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:50:03.265419 17598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:03.265611 17583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:50:03.265697 17583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "41685603f78341fdb31565b6ed33e296"
format_stamp: "Formatted at 2026-06-20 14:50:03 on dist-test-slave-cx89"
server_key: "72b43c117789b92d5e0beec780e71fb4"
server_key_iv: "290fa3c2710009f4654eff4acc7fb2e5"
server_key_version: "encryptionkey@0"
I20260620 14:50:03.265764 17583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:03.282476 17583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:03.284395 17583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:03.284525 17583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:03.284734 17583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:03.285017 17583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:03.285048 17583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:03.285076 17583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:03.285094 17583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:03.290372 17583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38321
I20260620 14:50:03.290413 17711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38321 every 8 connection(s)
I20260620 14:50:03.290704 17583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:50:03.295825 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 17583
I20260620 14:50:03.295953 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:50:03.296167 392 external_mini_cluster.cc:1428] Setting key 589e163b5da393077421c4edaacd359e
Jun 20 14:50:03 dist-test-slave-cx89 krb5kdc[17342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:50:03.300064 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.291341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55427 (local address 127.0.98.62:40297)
0620 14:50:03.291438 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:03.291440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:03.291657 (+ 217us) server_negotiation.cc:407] Connection header received
0620 14:50:03.292560 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:03.292561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:03.292571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:03.292621 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:03.293201 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.293702 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.294740 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.294939 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.295004 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.297503 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:03.297504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:03.297504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:03.297511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:03.298755 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:03.299126 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:03.299128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:03.299129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:03.299182 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:03.299396 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:03.299397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:03.299397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:03.299533 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:03.299590 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:03.299886 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 14:50:03.299976 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:50:03.300231 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.291321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40297 (local address 127.0.98.2:55427)
0620 14:50:03.291538 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:03.291577 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:03.292459 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:03.292680 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:03.292684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:03.292933 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:03.293124 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:03.293128 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.293777 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.293778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:03.294658 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:03.294661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.295052 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.295053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:03.295175 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.295783 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:03.295794 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:03.297416 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:03.298831 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:03.298838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:03.298841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:03.299054 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:03.299234 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:03.299235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:03.299235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:03.299341 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:03.299586 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:03.299591 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:03.299753 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 14:50:03.299840 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:50:03.300014 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":96,"threads_started":1}
I20260620 14:50:03.300644 17712 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40297
I20260620 14:50:03.300730 17712 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:03.300896 17712 heartbeater.cc:511] Master 127.0.98.62:40297 requested a full tablet report, sending...
I20260620 14:50:03.301278 17387 ts_manager.cc:194] Registered new tserver with Master: 41685603f78341fdb31565b6ed33e296 (127.0.98.2:38321)
I20260620 14:50:03.301774 17387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55427
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:50:03.320255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40297
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:03.399251 17717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:03.399487 17717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:03.399516 17717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:03.400696 17717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:03.400740 17717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:03.400765 17717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:50:03.402266 17717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:03.402451 17717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:03.402629 17717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:03.407407 17725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:03.407517 17717 server_base.cc:1034] running on GCE node
W20260620 14:50:03.407467 17723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:03.407482 17722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:03.407868 17717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:03.408085 17717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:03.409250 17717 hybrid_clock.cc:648] HybridClock initialized: now 1781967003409238 us; error 28 us; skew 500 ppm
Jun 20 14:50:03 dist-test-slave-cx89 krb5kdc[17342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:03.411670 17717 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:50:03.412101 17717 webserver.cc:460] Webserver started at http://127.0.98.3:41449/ using document root <none> and password file <none>
I20260620 14:50:03.412302 17717 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:03.412348 17717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:03.412472 17717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:03.413512 17717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "0d4f146eb09e40399cd10da3b0c3a76f"
format_stamp: "Formatted at 2026-06-20 14:50:03 on dist-test-slave-cx89"
server_key: "5ff20c813df60cc84e98d66637cf3777"
server_key_iv: "03b09d1cba31606eb0d2cdba389f565b"
server_key_version: "encryptionkey@0"
I20260620 14:50:03.413834 17717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0d4f146eb09e40399cd10da3b0c3a76f"
format_stamp: "Formatted at 2026-06-20 14:50:03 on dist-test-slave-cx89"
server_key: "5ff20c813df60cc84e98d66637cf3777"
server_key_iv: "03b09d1cba31606eb0d2cdba389f565b"
server_key_version: "encryptionkey@0"
I20260620 14:50:03.415035 17717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:50:03.415760 17732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:03.415904 17717 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:03.415987 17717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "0d4f146eb09e40399cd10da3b0c3a76f"
format_stamp: "Formatted at 2026-06-20 14:50:03 on dist-test-slave-cx89"
server_key: "5ff20c813df60cc84e98d66637cf3777"
server_key_iv: "03b09d1cba31606eb0d2cdba389f565b"
server_key_version: "encryptionkey@0"
I20260620 14:50:03.416046 17717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:03.448218 17717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:03.450407 17717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:03.450547 17717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:03.450771 17717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:03.451077 17717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:03.451110 17717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:03.451140 17717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:03.451175 17717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:03.456712 17717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35417
I20260620 14:50:03.456766 17845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35417 every 8 connection(s)
I20260620 14:50:03.457050 17717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:50:03 dist-test-slave-cx89 krb5kdc[17342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:50:03.464843 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 17717
I20260620 14:50:03.464938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:50:03.465143 392 external_mini_cluster.cc:1428] Setting key 75d826ab17dc26e264b2fc4c1de51d5d
I20260620 14:50:03.466132 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.457745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53743 (local address 127.0.98.62:40297)
0620 14:50:03.457805 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:03.457807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:03.457986 (+ 179us) server_negotiation.cc:407] Connection header received
0620 14:50:03.458908 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:03.458909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:03.458922 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:03.458974 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:03.459626 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.460118 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.461105 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.461298 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.461352 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.463352 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:03.463354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:03.463354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:03.463362 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:03.464635 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:03.465052 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:03.465053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:03.465054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:03.465125 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:03.465443 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:03.465445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:03.465445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:03.465584 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:03.465650 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:03.465962 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 14:50:03.466044 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:50:03.466316 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.457702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40297 (local address 127.0.98.3:53743)
0620 14:50:03.457872 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:03.457909 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:03.458804 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:03.459045 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:03.459049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:03.459302 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:03.459544 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:03.459549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.460200 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.460201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:03.461022 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:03.461024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.461364 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.461365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:03.461494 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.461964 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:03.461973 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:03.463268 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:03.464726 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:03.464732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:03.464734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:03.464941 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:03.465240 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:03.465241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:03.465242 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:03.465377 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:03.465651 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:03.465657 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:03.465825 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 14:50:03.465931 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:50:03.466101 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260620 14:50:03.466743 17846 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40297
I20260620 14:50:03.466833 17846 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:03.466976 17846 heartbeater.cc:511] Master 127.0.98.62:40297 requested a full tablet report, sending...
I20260620 14:50:03.467304 17387 ts_manager.cc:194] Registered new tserver with Master: 0d4f146eb09e40399cd10da3b0c3a76f (127.0.98.3:35417)
I20260620 14:50:03.467674 17387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53743
I20260620 14:50:03.476773 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:50:03.483703 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.478012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.0.98.62:40297)
0620 14:50:03.478070 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:03.478072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:03.478124 (+ 52us) server_negotiation.cc:407] Connection header received
0620 14:50:03.478197 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:03.478198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:03.478207 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:03.478253 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:03.478776 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.479261 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.480109 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.480299 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.480352 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.481434 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:03.481435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:03.481436 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:03.481442 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:03.482666 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:03.482958 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:03.482959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:03.482960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:03.483009 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:03.483167 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:03.483168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:03.483168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:03.483315 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:03.483401 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:03.483539 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 14:50:03.483614 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:50:03.485139 17387 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:50:03.486689 17387 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:50:03.492856 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.488904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38321 (local address 127.0.0.1:46072)
0620 14:50:03.489257 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:03.489267 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:03.489317 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:03.489757 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:03.489758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:03.489761 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:03.489797 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:03.489799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.491004 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.491005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:03.491887 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:03.491889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.492540 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.492541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:03.492625 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.492639 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 14:50:03.492711 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:50:03.492765 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":2,"thread_start_us":42,"threads_started":1}
I20260620 14:50:03.492851 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.489097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46072 (local address 127.0.98.2:38321)
0620 14:50:03.489600 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:03.489604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:03.489613 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:50:03.489648 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:03.489650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:03.489664 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:03.489862 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:03.489939 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.490921 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.491959 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.492465 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.492528 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.492590 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:03.492733 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:50:03.492772 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":95,"threads_started":1}
I20260620 14:50:03.493044 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.488912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35417 (local address 127.0.0.1:60386)
0620 14:50:03.489372 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:03.489379 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:03.489402 (+ 23us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:03.489632 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:03.489633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:03.489635 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:03.489694 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:03.489695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.490594 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.490595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:03.491919 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:03.491921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.492831 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.492832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:03.492893 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.492897 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:50:03.492925 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 14:50:03.492965 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":16,"thread_start_us":85,"threads_started":1}
I20260620 14:50:03.493186 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.488904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41337 (local address 127.0.0.1:40016)
0620 14:50:03.489052 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:03.489093 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:03.489158 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:03.489454 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:03.489457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:03.489460 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:03.489530 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:03.489532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.490816 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.490817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:03.491919 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:03.491921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.492886 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.492887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:03.492937 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.492940 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:50:03.492982 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 14:50:03.493015 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260620 14:50:03.493304 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.489090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.0.98.1:41337)
0620 14:50:03.489312 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:03.489316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:03.489325 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:50:03.489358 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:03.489359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:03.489371 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:03.489438 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:03.489611 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.490714 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.492016 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.492738 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.492802 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.492858 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:03.493084 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 14:50:03.493230 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
I20260620 14:50:03.493389 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.489152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60386 (local address 127.0.98.3:35417)
0620 14:50:03.489481 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:03.489485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:03.489492 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:50:03.489525 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:03.489527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:03.489543 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:03.489768 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:03.489849 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.490495 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.492016 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.492734 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.493177 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.493248 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:03.493272 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 14:50:03.493314 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":72,"threads_started":1}
I20260620 14:50:03.493983 17780 tablet_service.cc:1467] Processing CreateTablet for tablet af00abbf71034e568d934145a88393c3 (DEFAULT_TABLE table=test-table [id=c7c763b436bc4c28913a8871e90d8ea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:03.493985 17646 tablet_service.cc:1467] Processing CreateTablet for tablet af00abbf71034e568d934145a88393c3 (DEFAULT_TABLE table=test-table [id=c7c763b436bc4c28913a8871e90d8ea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:03.493985 17512 tablet_service.cc:1467] Processing CreateTablet for tablet af00abbf71034e568d934145a88393c3 (DEFAULT_TABLE table=test-table [id=c7c763b436bc4c28913a8871e90d8ea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:03.494300 17780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af00abbf71034e568d934145a88393c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:03.494307 17512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af00abbf71034e568d934145a88393c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:03.494307 17646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af00abbf71034e568d934145a88393c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:03.496701 17863 tablet_bootstrap.cc:492] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f: Bootstrap starting.
I20260620 14:50:03.497335 17863 tablet_bootstrap.cc:654] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:03.497537 17864 tablet_bootstrap.cc:492] T af00abbf71034e568d934145a88393c3 P 511cb13ae334492bbcda9f6f78b5b633: Bootstrap starting.
I20260620 14:50:03.497774 17865 tablet_bootstrap.cc:492] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296: Bootstrap starting.
I20260620 14:50:03.498018 17863 log.cc:824] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:03.498157 17864 tablet_bootstrap.cc:654] T af00abbf71034e568d934145a88393c3 P 511cb13ae334492bbcda9f6f78b5b633: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:03.498332 17865 tablet_bootstrap.cc:654] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:03.498633 17864 log.cc:824] T af00abbf71034e568d934145a88393c3 P 511cb13ae334492bbcda9f6f78b5b633: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:03.498808 17863 tablet_bootstrap.cc:492] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f: No bootstrap required, opened a new log
I20260620 14:50:03.498816 17865 log.cc:824] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:03.498874 17863 ts_tablet_manager.cc:1397] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:50:03.499259 17864 tablet_bootstrap.cc:492] T af00abbf71034e568d934145a88393c3 P 511cb13ae334492bbcda9f6f78b5b633: No bootstrap required, opened a new log
I20260620 14:50:03.499315 17864 ts_tablet_manager.cc:1397] T af00abbf71034e568d934145a88393c3 P 511cb13ae334492bbcda9f6f78b5b633: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:50:03.499509 17865 tablet_bootstrap.cc:492] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296: No bootstrap required, opened a new log
I20260620 14:50:03.499656 17865 ts_tablet_manager.cc:1397] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:50:03.500789 17863 raft_consensus.cc:348] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511cb13ae334492bbcda9f6f78b5b633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41337 } } peers { permanent_uuid: "41685603f78341fdb31565b6ed33e296" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38321 } } peers { permanent_uuid: "0d4f146eb09e40399cd10da3b0c3a76f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35417 } }
I20260620 14:50:03.500965 17863 raft_consensus.cc:374] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:03.500881 17864 raft_consensus.cc:348] T af00abbf71034e568d934145a88393c3 P 511cb13ae334492bbcda9f6f78b5b633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511cb13ae334492bbcda9f6f78b5b633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41337 } } peers { permanent_uuid: "41685603f78341fdb31565b6ed33e296" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38321 } } peers { permanent_uuid: "0d4f146eb09e40399cd10da3b0c3a76f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35417 } }
I20260620 14:50:03.501011 17863 raft_consensus.cc:729] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d4f146eb09e40399cd10da3b0c3a76f, State: Initialized, Role: FOLLOWER
I20260620 14:50:03.501020 17864 raft_consensus.cc:374] T af00abbf71034e568d934145a88393c3 P 511cb13ae334492bbcda9f6f78b5b633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:03.501053 17864 raft_consensus.cc:729] T af00abbf71034e568d934145a88393c3 P 511cb13ae334492bbcda9f6f78b5b633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 511cb13ae334492bbcda9f6f78b5b633, State: Initialized, Role: FOLLOWER
I20260620 14:50:03.501123 17863 consensus_queue.cc:260] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511cb13ae334492bbcda9f6f78b5b633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41337 } } peers { permanent_uuid: "41685603f78341fdb31565b6ed33e296" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38321 } } peers { permanent_uuid: "0d4f146eb09e40399cd10da3b0c3a76f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35417 } }
I20260620 14:50:03.501152 17864 consensus_queue.cc:260] T af00abbf71034e568d934145a88393c3 P 511cb13ae334492bbcda9f6f78b5b633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511cb13ae334492bbcda9f6f78b5b633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41337 } } peers { permanent_uuid: "41685603f78341fdb31565b6ed33e296" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38321 } } peers { permanent_uuid: "0d4f146eb09e40399cd10da3b0c3a76f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35417 } }
I20260620 14:50:03.501381 17864 ts_tablet_manager.cc:1428] T af00abbf71034e568d934145a88393c3 P 511cb13ae334492bbcda9f6f78b5b633: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:50:03.501399 17846 heartbeater.cc:503] Master 127.0.98.62:40297 was elected leader, sending a full tablet report...
I20260620 14:50:03.501381 17863 ts_tablet_manager.cc:1428] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:50:03.501632 17578 heartbeater.cc:503] Master 127.0.98.62:40297 was elected leader, sending a full tablet report...
I20260620 14:50:03.501762 17865 raft_consensus.cc:348] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511cb13ae334492bbcda9f6f78b5b633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41337 } } peers { permanent_uuid: "41685603f78341fdb31565b6ed33e296" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38321 } } peers { permanent_uuid: "0d4f146eb09e40399cd10da3b0c3a76f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35417 } }
I20260620 14:50:03.501961 17865 raft_consensus.cc:374] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:03.502009 17865 raft_consensus.cc:729] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41685603f78341fdb31565b6ed33e296, State: Initialized, Role: FOLLOWER
I20260620 14:50:03.502153 17865 consensus_queue.cc:260] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511cb13ae334492bbcda9f6f78b5b633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41337 } } peers { permanent_uuid: "41685603f78341fdb31565b6ed33e296" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38321 } } peers { permanent_uuid: "0d4f146eb09e40399cd10da3b0c3a76f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35417 } }
I20260620 14:50:03.502417 17712 heartbeater.cc:503] Master 127.0.98.62:40297 was elected leader, sending a full tablet report...
I20260620 14:50:03.502424 17865 ts_tablet_manager.cc:1428] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260620 14:50:03.541285 17713 tablet.cc:2369] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:50:03.618674 17871 raft_consensus.cc:482] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:50:03.618795 17871 raft_consensus.cc:504] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511cb13ae334492bbcda9f6f78b5b633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41337 } } peers { permanent_uuid: "41685603f78341fdb31565b6ed33e296" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38321 } } peers { permanent_uuid: "0d4f146eb09e40399cd10da3b0c3a76f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35417 } }
I20260620 14:50:03.619208 17871 leader_election.cc:288] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 511cb13ae334492bbcda9f6f78b5b633 (127.0.98.1:41337), 0d4f146eb09e40399cd10da3b0c3a76f (127.0.98.3:35417)
I20260620 14:50:03.621980 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.619489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51297 (local address 127.0.98.3:35417)
0620 14:50:03.619554 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:03.619556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:03.619704 (+ 148us) server_negotiation.cc:407] Connection header received
0620 14:50:03.619748 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:03.619749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:03.619758 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:03.619808 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:03.619977 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.620439 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.621316 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.621688 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.621751 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.621761 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:03.621875 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 14:50:03.621909 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:50:03.621992 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.619449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51245 (local address 127.0.98.1:41337)
0620 14:50:03.619512 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:03.619514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:03.619548 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:50:03.619586 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:03.619587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:03.619599 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:03.619648 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:03.619824 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.620349 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.621294 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.621661 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.621711 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.621720 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:03.621880 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 14:50:03.621916 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:50:03.622004 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.619379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41337 (local address 127.0.98.2:51245)
0620 14:50:03.619452 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:03.619463 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:03.619511 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:03.619667 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:03.619668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:03.619670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:03.619747 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:03.619749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.620438 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.620439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:03.621209 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:03.621211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.621733 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.621734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:03.621813 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.621816 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:50:03.621862 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 14:50:03.621901 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 14:50:03.622013 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.619379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35417 (local address 127.0.98.2:51297)
0620 14:50:03.619659 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:03.619667 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:03.619708 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:03.619822 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:03.619822 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:03.619824 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:03.619889 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:03.619891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.620520 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.620521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:03.621239 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:03.621240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.621762 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.621763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:03.621816 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.621819 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:50:03.621863 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 14:50:03.621904 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":104,"thread_start_us":110,"threads_started":1}
I20260620 14:50:03.622390 17800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af00abbf71034e568d934145a88393c3" candidate_uuid: "41685603f78341fdb31565b6ed33e296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d4f146eb09e40399cd10da3b0c3a76f" is_pre_election: true
I20260620 14:50:03.622368 17532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af00abbf71034e568d934145a88393c3" candidate_uuid: "41685603f78341fdb31565b6ed33e296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "511cb13ae334492bbcda9f6f78b5b633" is_pre_election: true
I20260620 14:50:03.622525 17800 raft_consensus.cc:2454] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41685603f78341fdb31565b6ed33e296 in term 0.
I20260620 14:50:03.622540 17532 raft_consensus.cc:2454] T af00abbf71034e568d934145a88393c3 P 511cb13ae334492bbcda9f6f78b5b633 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41685603f78341fdb31565b6ed33e296 in term 0.
I20260620 14:50:03.622731 17602 leader_election.cc:302] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d4f146eb09e40399cd10da3b0c3a76f, 41685603f78341fdb31565b6ed33e296; no voters:
I20260620 14:50:03.622900 17871 raft_consensus.cc:2781] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:50:03.622947 17871 raft_consensus.cc:482] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:50:03.622977 17871 raft_consensus.cc:3037] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:03.623688 17871 raft_consensus.cc:504] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511cb13ae334492bbcda9f6f78b5b633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41337 } } peers { permanent_uuid: "41685603f78341fdb31565b6ed33e296" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38321 } } peers { permanent_uuid: "0d4f146eb09e40399cd10da3b0c3a76f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35417 } }
I20260620 14:50:03.623809 17871 leader_election.cc:288] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [CANDIDATE]: Term 1 election: Requested vote from peers 511cb13ae334492bbcda9f6f78b5b633 (127.0.98.1:41337), 0d4f146eb09e40399cd10da3b0c3a76f (127.0.98.3:35417)
I20260620 14:50:03.623955 17532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af00abbf71034e568d934145a88393c3" candidate_uuid: "41685603f78341fdb31565b6ed33e296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "511cb13ae334492bbcda9f6f78b5b633"
I20260620 14:50:03.624003 17800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af00abbf71034e568d934145a88393c3" candidate_uuid: "41685603f78341fdb31565b6ed33e296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d4f146eb09e40399cd10da3b0c3a76f"
I20260620 14:50:03.624040 17532 raft_consensus.cc:3037] T af00abbf71034e568d934145a88393c3 P 511cb13ae334492bbcda9f6f78b5b633 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:03.624066 17800 raft_consensus.cc:3037] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:03.624864 17532 raft_consensus.cc:2454] T af00abbf71034e568d934145a88393c3 P 511cb13ae334492bbcda9f6f78b5b633 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41685603f78341fdb31565b6ed33e296 in term 1.
I20260620 14:50:03.624864 17800 raft_consensus.cc:2454] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41685603f78341fdb31565b6ed33e296 in term 1.
I20260620 14:50:03.625017 17600 leader_election.cc:302] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41685603f78341fdb31565b6ed33e296, 511cb13ae334492bbcda9f6f78b5b633; no voters:
I20260620 14:50:03.625118 17871 raft_consensus.cc:2781] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:03.625255 17871 raft_consensus.cc:686] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [term 1 LEADER]: Becoming Leader. State: Replica: 41685603f78341fdb31565b6ed33e296, State: Running, Role: LEADER
I20260620 14:50:03.625340 17871 consensus_queue.cc:237] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511cb13ae334492bbcda9f6f78b5b633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41337 } } peers { permanent_uuid: "41685603f78341fdb31565b6ed33e296" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38321 } } peers { permanent_uuid: "0d4f146eb09e40399cd10da3b0c3a76f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35417 } }
I20260620 14:50:03.626142 17386 catalog_manager.cc:5495] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41685603f78341fdb31565b6ed33e296 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "41685603f78341fdb31565b6ed33e296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "511cb13ae334492bbcda9f6f78b5b633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41685603f78341fdb31565b6ed33e296" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d4f146eb09e40399cd10da3b0c3a76f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20260620 14:50:03.639129 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.637103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46074 (local address 127.0.98.2:38321)
0620 14:50:03.637153 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:03.637155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:03.637163 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:50:03.637194 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:03.637195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:03.637210 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:03.637265 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:03.637351 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.637796 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.638499 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.638698 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.638752 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.638878 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:03.638974 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:03.639027 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:03.639039 (+ 12us) server_negotiation.cc:299] Negotiation successful
0620 14:50:03.639053 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:50:03.640903 17532 raft_consensus.cc:1261] T af00abbf71034e568d934145a88393c3 P 511cb13ae334492bbcda9f6f78b5b633 [term 1 FOLLOWER]: Refusing update from remote peer 41685603f78341fdb31565b6ed33e296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:03.640903 17800 raft_consensus.cc:1261] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f [term 1 FOLLOWER]: Refusing update from remote peer 41685603f78341fdb31565b6ed33e296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:03.641287 17873 consensus_queue.cc:1035] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d4f146eb09e40399cd10da3b0c3a76f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:03.641419 17871 consensus_queue.cc:1035] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "511cb13ae334492bbcda9f6f78b5b633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:50:03.641541 17579 tablet.cc:2369] T af00abbf71034e568d934145a88393c3 P 511cb13ae334492bbcda9f6f78b5b633: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:50:03.644642 17877 mvcc.cc:205] Tried to move back new op lower bound from 7298936846910349312 to 7298936846849916928. Current Snapshot: MvccSnapshot[applied={T|T < 7298936846910349312}]
I20260620 14:50:03.645727 17387 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:03.645824 17387 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:03.646585 17387 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23fe64bc691e4d9ca57979d0e10cd3cd: Sending DeleteTablet for 3 replicas of tablet af00abbf71034e568d934145a88393c3
I20260620 14:50:03.646937 17780 tablet_service.cc:1514] Processing DeleteTablet for tablet af00abbf71034e568d934145a88393c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60386
I20260620 14:50:03.646916 17512 tablet_service.cc:1514] Processing DeleteTablet for tablet af00abbf71034e568d934145a88393c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40016
I20260620 14:50:03.646932 17646 tablet_service.cc:1514] Processing DeleteTablet for tablet af00abbf71034e568d934145a88393c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46072
I20260620 14:50:03.647176 17885 tablet_replica.cc:330] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f: stopping tablet replica
I20260620 14:50:03.647323 17886 tablet_replica.cc:330] T af00abbf71034e568d934145a88393c3 P 511cb13ae334492bbcda9f6f78b5b633: stopping tablet replica
I20260620 14:50:03.647321 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 17449
I20260620 14:50:03.648598 17887 tablet_replica.cc:330] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296: stopping tablet replica
I20260620 14:50:03.648761 17885 raft_consensus.cc:2229] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:03.648916 17887 raft_consensus.cc:2229] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:50:03.649065 17885 raft_consensus.cc:2258] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:03.649223 17887 raft_consensus.cc:2258] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:03.649788 17885 ts_tablet_manager.cc:1905] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:03.649792 17887 ts_tablet_manager.cc:1905] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:03.651579 17885 ts_tablet_manager.cc:1918] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:03.651597 17887 ts_tablet_manager.cc:1918] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:03.651659 17887 log.cc:1196] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/af00abbf71034e568d934145a88393c3
I20260620 14:50:03.651719 17885 log.cc:1196] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/af00abbf71034e568d934145a88393c3
I20260620 14:50:03.652038 17885 ts_tablet_manager.cc:1939] T af00abbf71034e568d934145a88393c3 P 0d4f146eb09e40399cd10da3b0c3a76f: Deleting consensus metadata
I20260620 14:50:03.652048 17887 ts_tablet_manager.cc:1939] T af00abbf71034e568d934145a88393c3 P 41685603f78341fdb31565b6ed33e296: Deleting consensus metadata
W20260620 14:50:03.652091 17372 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41337 (error 108)
I20260620 14:50:03.652442 17374 catalog_manager.cc:4841] TS 0d4f146eb09e40399cd10da3b0c3a76f (127.0.98.3:35417): tablet af00abbf71034e568d934145a88393c3 (table test-table [id=c7c763b436bc4c28913a8871e90d8ea3]) successfully deleted
I20260620 14:50:03.652442 17371 catalog_manager.cc:4841] TS 41685603f78341fdb31565b6ed33e296 (127.0.98.2:38321): tablet af00abbf71034e568d934145a88393c3 (table test-table [id=c7c763b436bc4c28913a8871e90d8ea3]) successfully deleted
I20260620 14:50:03.652724 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.652567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41337 (local address 127.0.0.1:40026)
0620 14:50:03.652606 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:03.652643 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260620 14:50:03.652853 17372 catalog_manager.cc:4584] TS 511cb13ae334492bbcda9f6f78b5b633 (127.0.98.1:41337): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af00abbf71034e568d934145a88393c3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41337: connect: Connection refused (error 111)
I20260620 14:50:03.653173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 17583
I20260620 14:50:03.657892 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 17717
I20260620 14:50:03.661974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 17355
2026-06-20T14:50:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1934 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T14:50:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:03Z Disabled control of system clock
I20260620 14:50:03.687023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45019
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:50:03.765233 17892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:03.765445 17892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:03.765466 17892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:03.766577 17892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:03.766620 17892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:03.766633 17892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:03.766644 17892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:03.766654 17892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:03.767999 17892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45019
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:03.768201 17892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:03.768437 17892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:03.772884 17900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:03.772923 17898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:03.772962 17897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:03.773133 17892 server_base.cc:1034] running on GCE node
I20260620 14:50:03.773352 17892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:03.773597 17892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:03.774734 17892 hybrid_clock.cc:648] HybridClock initialized: now 1781967003774718 us; error 28 us; skew 500 ppm
I20260620 14:50:03.775915 17892 webserver.cc:460] Webserver started at http://127.0.98.62:33995/ using document root <none> and password file <none>
I20260620 14:50:03.776129 17892 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:03.776177 17892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:03.776298 17892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:03.777303 17892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "fe4ab17c12a546dc90e3bdc460e5ca9e"
format_stamp: "Formatted at 2026-06-20 14:50:03 on dist-test-slave-cx89"
server_key: "a415dfba0ac6c13c59a1255b055f5fb5"
server_key_iv: "6783fd2bff3b430abd4c4c60a871726d"
server_key_version: "encryptionkey@0"
I20260620 14:50:03.777619 17892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "fe4ab17c12a546dc90e3bdc460e5ca9e"
format_stamp: "Formatted at 2026-06-20 14:50:03 on dist-test-slave-cx89"
server_key: "a415dfba0ac6c13c59a1255b055f5fb5"
server_key_iv: "6783fd2bff3b430abd4c4c60a871726d"
server_key_version: "encryptionkey@0"
I20260620 14:50:03.778757 17892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:50:03.779492 17906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:03.779650 17892 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:03.779731 17892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "fe4ab17c12a546dc90e3bdc460e5ca9e"
format_stamp: "Formatted at 2026-06-20 14:50:03 on dist-test-slave-cx89"
server_key: "a415dfba0ac6c13c59a1255b055f5fb5"
server_key_iv: "6783fd2bff3b430abd4c4c60a871726d"
server_key_version: "encryptionkey@0"
I20260620 14:50:03.779796 17892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:03.800788 17892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:03.801102 17892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:03.801240 17892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:03.804848 17892 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45019
I20260620 14:50:03.804886 17958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45019 every 8 connection(s)
I20260620 14:50:03.805208 17892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:03.806167 17959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:03.808949 17959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e: Bootstrap starting.
I20260620 14:50:03.809602 17959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:03.809904 17959 log.cc:824] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:03.810547 17959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e: No bootstrap required, opened a new log
I20260620 14:50:03.810889 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 17892
I20260620 14:50:03.810994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:03.811188 392 external_mini_cluster.cc:1428] Setting key 8e3ff59020eceb16738b0f712f75759f
I20260620 14:50:03.812556 17959 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4ab17c12a546dc90e3bdc460e5ca9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } }
I20260620 14:50:03.812697 17959 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:03.812731 17959 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe4ab17c12a546dc90e3bdc460e5ca9e, State: Initialized, Role: FOLLOWER
I20260620 14:50:03.812888 17959 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4ab17c12a546dc90e3bdc460e5ca9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } }
I20260620 14:50:03.812975 17959 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:03.813004 17959 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:03.813051 17959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:03.813550 17959 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4ab17c12a546dc90e3bdc460e5ca9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } }
I20260620 14:50:03.813634 17959 leader_election.cc:302] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe4ab17c12a546dc90e3bdc460e5ca9e; no voters:
I20260620 14:50:03.813791 17959 leader_election.cc:288] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:03.813832 17964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:03.814034 17964 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e [term 1 LEADER]: Becoming Leader. State: Replica: fe4ab17c12a546dc90e3bdc460e5ca9e, State: Running, Role: LEADER
I20260620 14:50:03.814047 17959 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:03.814743 17964 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4ab17c12a546dc90e3bdc460e5ca9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } }
I20260620 14:50:03.815354 17965 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe4ab17c12a546dc90e3bdc460e5ca9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4ab17c12a546dc90e3bdc460e5ca9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } }
I20260620 14:50:03.815351 17966 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe4ab17c12a546dc90e3bdc460e5ca9e. Latest consensus state: current_term: 1 leader_uuid: "fe4ab17c12a546dc90e3bdc460e5ca9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4ab17c12a546dc90e3bdc460e5ca9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } }
W20260620 14:50:03.815958 17978 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:50:03.816184 17978 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:50:03.816159 17965 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:03.816167 17966 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:03.816371 17981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:03.816524 17981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:03.817818 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.811753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58278 (local address 127.0.98.62:45019)
0620 14:50:03.811954 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:03.811962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:03.811988 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:50:03.812524 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:03.812530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:03.812587 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:03.812709 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:03.812949 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.814423 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.815528 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.816125 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.816246 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.817179 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:03.817183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:03.817183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:03.817378 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:03.817422 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:03.817436 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:03.817500 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:03.817566 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:50:03.817644 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260620 14:50:03.819864 17981 catalog_manager.cc:1347] Generated new cluster ID: 0fc07caeb73f4e6e9927c5f4e7077d9e
I20260620 14:50:03.819926 17981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:03.837217 17981 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:03.837901 17981 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:03.844458 17981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e: Generated new TSK 0
I20260620 14:50:03.844660 17981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:03.849128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45019
--builtin_ntp_servers=127.0.98.20:35779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:50:03.932566 17983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:03.932773 17983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:03.932804 17983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:03.933919 17983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:03.933959 17983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:03.933992 17983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:50:03.935316 17983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:03.935563 17983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:03.935828 17983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:03.940688 17988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:03.940719 17991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:03.940688 17989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:03.941009 17983 server_base.cc:1034] running on GCE node
I20260620 14:50:03.941217 17983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:03.941433 17983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:03.942580 17983 hybrid_clock.cc:648] HybridClock initialized: now 1781967003942564 us; error 30 us; skew 500 ppm
I20260620 14:50:03.944044 17983 webserver.cc:460] Webserver started at http://127.0.98.1:45777/ using document root <none> and password file <none>
I20260620 14:50:03.944334 17983 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:03.944402 17983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:03.944546 17983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:03.945909 17983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "2fa08f9389b84169892b6ff041ff687d"
format_stamp: "Formatted at 2026-06-20 14:50:03 on dist-test-slave-cx89"
server_key: "645014a8a59b1eae9a30322c365a74a8"
server_key_iv: "7a0698363b4119fd70717b9c68de55ac"
server_key_version: "encryptionkey@0"
I20260620 14:50:03.946322 17983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2fa08f9389b84169892b6ff041ff687d"
format_stamp: "Formatted at 2026-06-20 14:50:03 on dist-test-slave-cx89"
server_key: "645014a8a59b1eae9a30322c365a74a8"
server_key_iv: "7a0698363b4119fd70717b9c68de55ac"
server_key_version: "encryptionkey@0"
I20260620 14:50:03.947999 17983 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260620 14:50:03.948849 17997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:03.949005 17983 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:03.949097 17983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "2fa08f9389b84169892b6ff041ff687d"
format_stamp: "Formatted at 2026-06-20 14:50:03 on dist-test-slave-cx89"
server_key: "645014a8a59b1eae9a30322c365a74a8"
server_key_iv: "7a0698363b4119fd70717b9c68de55ac"
server_key_version: "encryptionkey@0"
I20260620 14:50:03.949167 17983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:03.966413 17983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:03.966836 17983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:03.966952 17983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:03.967136 17983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:03.967487 17983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:03.967523 17983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:03.967555 17983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:03.967571 17983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:03.972968 17983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32863
I20260620 14:50:03.973028 18110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32863 every 8 connection(s)
I20260620 14:50:03.973346 17983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:50:03.974187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 17983
I20260620 14:50:03.974268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:50:03.974452 392 external_mini_cluster.cc:1428] Setting key 4e7a3e828fb13484b01a18061c705e82
I20260620 14:50:03.975322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45019
--builtin_ntp_servers=127.0.98.20:35779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:50:03.979069 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.974023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45643 (local address 127.0.98.62:45019)
0620 14:50:03.974095 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:03.974097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:03.974375 (+ 278us) server_negotiation.cc:407] Connection header received
0620 14:50:03.974962 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:03.974963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:03.974976 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:03.975036 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:03.975889 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.976504 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.977627 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.977827 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.977887 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.978823 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:03.978825 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:03.978825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:03.978829 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:03.978844 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:03.978848 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:03.978883 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:03.978972 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 14:50:03.979003 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:50:03.979319 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:03.974049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45019 (local address 127.0.98.1:45643)
0620 14:50:03.974242 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:03.974286 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:03.974862 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:03.975205 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:03.975209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:03.975217 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:03.975799 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:03.975805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.976583 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.976584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:03.977516 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:03.977519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:03.977903 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:03.977904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:03.978057 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:03.978653 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:50:03.978654 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:03.978696 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:50:03.978728 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:03.978898 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:03.978903 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:03.978906 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:50:03.978977 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:50:03.979092 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260620 14:50:03.979849 18111 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45019
I20260620 14:50:03.979939 18111 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:03.980118 18111 heartbeater.cc:511] Master 127.0.98.62:45019 requested a full tablet report, sending...
I20260620 14:50:03.980629 17923 ts_manager.cc:194] Registered new tserver with Master: 2fa08f9389b84169892b6ff041ff687d (127.0.98.1:32863)
I20260620 14:50:03.981410 17923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:45643
W20260620 14:50:04.057106 18114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:04.057317 18114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:04.057339 18114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:04.058487 18114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:04.058522 18114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:04.058547 18114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:50:04.059942 18114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:04.060168 18114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:04.060452 18114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:04.065425 18119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:04.065420 18122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:04.065428 18120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:04.065709 18114 server_base.cc:1034] running on GCE node
I20260620 14:50:04.065904 18114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:04.066165 18114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:04.067313 18114 hybrid_clock.cc:648] HybridClock initialized: now 1781967004067297 us; error 28 us; skew 500 ppm
I20260620 14:50:04.068586 18114 webserver.cc:460] Webserver started at http://127.0.98.2:41433/ using document root <none> and password file <none>
I20260620 14:50:04.068835 18114 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:04.068886 18114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:04.069017 18114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:04.070078 18114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "b1616d2721664fddb9596c61d9ee3e9c"
format_stamp: "Formatted at 2026-06-20 14:50:04 on dist-test-slave-cx89"
server_key: "9b6d37d7c824746192cb9737e95fb946"
server_key_iv: "65ce69bd758944ddb2ebc90a08f1f0fa"
server_key_version: "encryptionkey@0"
I20260620 14:50:04.070421 18114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b1616d2721664fddb9596c61d9ee3e9c"
format_stamp: "Formatted at 2026-06-20 14:50:04 on dist-test-slave-cx89"
server_key: "9b6d37d7c824746192cb9737e95fb946"
server_key_iv: "65ce69bd758944ddb2ebc90a08f1f0fa"
server_key_version: "encryptionkey@0"
I20260620 14:50:04.071667 18114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:50:04.072414 18128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:04.072582 18114 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:50:04.072659 18114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "b1616d2721664fddb9596c61d9ee3e9c"
format_stamp: "Formatted at 2026-06-20 14:50:04 on dist-test-slave-cx89"
server_key: "9b6d37d7c824746192cb9737e95fb946"
server_key_iv: "65ce69bd758944ddb2ebc90a08f1f0fa"
server_key_version: "encryptionkey@0"
I20260620 14:50:04.072727 18114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:04.083099 18114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:04.083384 18114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:04.083577 18114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:04.083794 18114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:04.084096 18114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:04.084129 18114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:04.084156 18114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:04.084173 18114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:04.089463 18114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42969
I20260620 14:50:04.089506 18241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42969 every 8 connection(s)
I20260620 14:50:04.089816 18114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:50:04.090034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 18114
I20260620 14:50:04.090143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:50:04.090317 392 external_mini_cluster.cc:1428] Setting key b1471dfde20e5e4bb8e1bd1dc375936c
I20260620 14:50:04.091320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45019
--builtin_ntp_servers=127.0.98.20:35779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:50:04.095209 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:04.090658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57993 (local address 127.0.98.62:45019)
0620 14:50:04.090702 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:04.090704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:04.090973 (+ 269us) server_negotiation.cc:407] Connection header received
0620 14:50:04.091532 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:04.091533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:04.091545 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:04.091588 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:04.092244 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.092739 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.093708 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.093962 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.094026 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:04.094922 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:04.094924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:04.094924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:04.094930 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:04.094949 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:04.094953 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:04.095008 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:04.095090 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:50:04.095121 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:50:04.095371 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:04.090662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45019 (local address 127.0.98.2:57993)
0620 14:50:04.090854 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:04.090889 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:04.091383 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:04.091661 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:04.091665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:04.091675 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:04.092162 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:04.092167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.092823 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.092824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:04.093628 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:04.093630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.094094 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.094095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:04.094229 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:04.094770 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:50:04.094771 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:04.094802 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:50:04.094827 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:04.095016 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:04.095019 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:04.095022 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:50:04.095077 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:50:04.095166 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20260620 14:50:04.095872 18242 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45019
I20260620 14:50:04.095961 18242 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:04.096141 18242 heartbeater.cc:511] Master 127.0.98.62:45019 requested a full tablet report, sending...
I20260620 14:50:04.096509 17923 ts_manager.cc:194] Registered new tserver with Master: b1616d2721664fddb9596c61d9ee3e9c (127.0.98.2:42969)
I20260620 14:50:04.096879 17923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:57993
W20260620 14:50:04.170100 18245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:04.170295 18245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:04.170323 18245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:04.171445 18245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:04.171481 18245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:04.171512 18245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:50:04.172824 18245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:04.173053 18245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:04.173314 18245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:04.177938 18250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:04.178040 18245 server_base.cc:1034] running on GCE node
W20260620 14:50:04.177981 18251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:04.177942 18253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:04.178339 18245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:04.178583 18245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:04.179713 18245 hybrid_clock.cc:648] HybridClock initialized: now 1781967004179698 us; error 27 us; skew 500 ppm
I20260620 14:50:04.180919 18245 webserver.cc:460] Webserver started at http://127.0.98.3:42051/ using document root <none> and password file <none>
I20260620 14:50:04.181133 18245 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:04.181181 18245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:04.181304 18245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:04.182351 18245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "555c0231e4aa41b497efbd807a156f96"
format_stamp: "Formatted at 2026-06-20 14:50:04 on dist-test-slave-cx89"
server_key: "500c2be52c233814a71337c79fd46f76"
server_key_iv: "cd29463c9eb4b4fed96bd6f8888e6896"
server_key_version: "encryptionkey@0"
I20260620 14:50:04.182677 18245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "555c0231e4aa41b497efbd807a156f96"
format_stamp: "Formatted at 2026-06-20 14:50:04 on dist-test-slave-cx89"
server_key: "500c2be52c233814a71337c79fd46f76"
server_key_iv: "cd29463c9eb4b4fed96bd6f8888e6896"
server_key_version: "encryptionkey@0"
I20260620 14:50:04.183917 18245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:50:04.184540 18259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:04.184689 18245 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:50:04.184768 18245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "555c0231e4aa41b497efbd807a156f96"
format_stamp: "Formatted at 2026-06-20 14:50:04 on dist-test-slave-cx89"
server_key: "500c2be52c233814a71337c79fd46f76"
server_key_iv: "cd29463c9eb4b4fed96bd6f8888e6896"
server_key_version: "encryptionkey@0"
I20260620 14:50:04.184824 18245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:04.194597 18245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:04.194852 18245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:04.194969 18245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:04.195158 18245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:04.195482 18245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:04.195516 18245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:04.195544 18245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:04.195583 18245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:04.200752 18245 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43133
I20260620 14:50:04.200803 18372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43133 every 8 connection(s)
I20260620 14:50:04.201081 18245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:50:04.205830 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:04.201681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40385 (local address 127.0.98.62:45019)
0620 14:50:04.201756 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:04.201758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:04.201927 (+ 169us) server_negotiation.cc:407] Connection header received
0620 14:50:04.202395 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:04.202396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:04.202404 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:04.202452 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:04.203037 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.203582 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.204463 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.204647 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.204700 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:04.205582 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:04.205584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:04.205584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:04.205589 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:04.205605 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:04.205609 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:04.205651 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:04.205722 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 14:50:04.205751 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:50:04.206020 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:04.201659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45019 (local address 127.0.98.3:40385)
0620 14:50:04.201820 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:04.201852 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:04.202320 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:04.202506 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:04.202509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:04.202517 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:04.202982 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:04.202986 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.203647 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.203647 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:04.204406 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:04.204408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.204728 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.204729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:04.204893 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:04.205444 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:50:04.205445 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:04.205473 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:50:04.205495 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:04.205657 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:04.205662 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:04.205666 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:50:04.205727 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:50:04.205824 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260620 14:50:04.206434 18373 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45019
I20260620 14:50:04.206513 18373 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:04.206523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 18245
I20260620 14:50:04.206602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:50:04.206696 18373 heartbeater.cc:511] Master 127.0.98.62:45019 requested a full tablet report, sending...
I20260620 14:50:04.206733 392 external_mini_cluster.cc:1428] Setting key 7a2601cf0609123e8d391dedb5fe455c
I20260620 14:50:04.207039 17923 ts_manager.cc:194] Registered new tserver with Master: 555c0231e4aa41b497efbd807a156f96 (127.0.98.3:43133)
I20260620 14:50:04.207391 17923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:40385
I20260620 14:50:04.207532 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:50:04.211045 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:04.208756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.0.98.62:45019)
0620 14:50:04.208814 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:04.208816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:04.208890 (+ 74us) server_negotiation.cc:407] Connection header received
0620 14:50:04.208953 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:04.208954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:04.208964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:04.208996 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:04.209123 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.209556 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.210207 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.210376 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.210418 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:04.210888 (+ 470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:04.210889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:04.210889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:04.210893 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:04.210905 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:04.210908 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:04.210942 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:04.210973 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 14:50:04.210991 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:50:04.212445 17923 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:50:04.213953 17923 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:50:04.219996 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:04.215827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42969 (local address 127.0.0.1:51980)
0620 14:50:04.216233 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:04.216268 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:04.216328 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:04.216494 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:04.216497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:04.216500 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:04.216563 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:04.216565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.217835 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.217836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:04.218888 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:04.218891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.219779 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.219780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:04.219831 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:04.219834 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:50:04.219872 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 14:50:04.219899 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":82,"threads_started":1}
I20260620 14:50:04.220012 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:04.216051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51980 (local address 127.0.98.2:42969)
0620 14:50:04.216211 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:04.216215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:04.216311 (+ 96us) server_negotiation.cc:407] Connection header received
0620 14:50:04.216387 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:04.216389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:04.216401 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:04.216468 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:04.216658 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.217757 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.218945 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.219702 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.219779 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:04.219866 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:04.219892 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 14:50:04.219929 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20260620 14:50:04.220060 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:04.215984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52078 (local address 127.0.98.1:32863)
0620 14:50:04.216134 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:04.216139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:04.216549 (+ 410us) server_negotiation.cc:407] Connection header received
0620 14:50:04.216599 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:04.216601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:04.216616 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:04.216684 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:04.216853 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.217528 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.218908 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.219520 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.219577 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:04.219672 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:04.219950 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 14:50:04.219987 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260620 14:50:04.219902 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:04.215828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32863 (local address 127.0.0.1:52078)
0620 14:50:04.216461 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:04.216469 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:04.216517 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:04.216753 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:04.216755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:04.216757 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:04.216799 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:04.216801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.217608 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.217609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:04.218825 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:04.218828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.219599 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.219600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:04.219677 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:04.219700 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:50:04.219770 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:50:04.219819 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":3,"thread_start_us":60,"threads_started":1}
I20260620 14:50:04.220703 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:04.216068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39884 (local address 127.0.98.3:43133)
0620 14:50:04.216426 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:04.216429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:04.216633 (+ 204us) server_negotiation.cc:407] Connection header received
0620 14:50:04.216734 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:04.216736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:04.216750 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:04.216797 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:04.216982 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.217961 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.219067 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.219754 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.219806 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:04.219872 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:04.220613 (+ 741us) server_negotiation.cc:299] Negotiation successful
0620 14:50:04.220644 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":81,"threads_started":1}
I20260620 14:50:04.220675 18176 tablet_service.cc:1467] Processing CreateTablet for tablet 375b88c14cf74ee2becb8a2c2eac1bc8 (DEFAULT_TABLE table=test-table [id=caabed09bb1342ce8cff6e3fab855823]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:04.220978 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:04.215827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43133 (local address 127.0.0.1:39884)
0620 14:50:04.216583 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:04.216590 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:04.216656 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:04.216868 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:04.216869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:04.216871 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:04.216913 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:04.216915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.218027 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.218028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:04.218983 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:04.218985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.219906 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.219907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:04.219966 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:04.219970 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:50:04.220778 (+ 808us) client_negotiation.cc:240] Negotiation successful
0620 14:50:04.220908 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":8,"thread_start_us":78,"threads_started":1}
I20260620 14:50:04.221045 18176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 375b88c14cf74ee2becb8a2c2eac1bc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:04.220990 18045 tablet_service.cc:1467] Processing CreateTablet for tablet 375b88c14cf74ee2becb8a2c2eac1bc8 (DEFAULT_TABLE table=test-table [id=caabed09bb1342ce8cff6e3fab855823]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:04.221309 18045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 375b88c14cf74ee2becb8a2c2eac1bc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:04.224344 18387 tablet_bootstrap.cc:492] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c: Bootstrap starting.
I20260620 14:50:04.224804 18307 tablet_service.cc:1467] Processing CreateTablet for tablet 375b88c14cf74ee2becb8a2c2eac1bc8 (DEFAULT_TABLE table=test-table [id=caabed09bb1342ce8cff6e3fab855823]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:04.224934 18387 tablet_bootstrap.cc:654] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:04.225064 18307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 375b88c14cf74ee2becb8a2c2eac1bc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:04.225200 18387 log.cc:824] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:04.225492 18388 tablet_bootstrap.cc:492] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d: Bootstrap starting.
I20260620 14:50:04.225988 18388 tablet_bootstrap.cc:654] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:04.226265 18388 log.cc:824] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:04.226789 18387 tablet_bootstrap.cc:492] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c: No bootstrap required, opened a new log
I20260620 14:50:04.226846 18387 ts_tablet_manager.cc:1397] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:50:04.226982 18388 tablet_bootstrap.cc:492] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d: No bootstrap required, opened a new log
I20260620 14:50:04.227052 18388 ts_tablet_manager.cc:1397] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:50:04.227957 18391 tablet_bootstrap.cc:492] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96: Bootstrap starting.
I20260620 14:50:04.228606 18391 tablet_bootstrap.cc:654] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:04.228950 18391 log.cc:824] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:04.229004 18388 raft_consensus.cc:348] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa08f9389b84169892b6ff041ff687d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "555c0231e4aa41b497efbd807a156f96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43133 } } peers { permanent_uuid: "b1616d2721664fddb9596c61d9ee3e9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42969 } }
I20260620 14:50:04.229135 18388 raft_consensus.cc:374] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:04.229158 18388 raft_consensus.cc:729] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa08f9389b84169892b6ff041ff687d, State: Initialized, Role: FOLLOWER
I20260620 14:50:04.229154 18387 raft_consensus.cc:348] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa08f9389b84169892b6ff041ff687d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "555c0231e4aa41b497efbd807a156f96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43133 } } peers { permanent_uuid: "b1616d2721664fddb9596c61d9ee3e9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42969 } }
I20260620 14:50:04.229254 18388 consensus_queue.cc:260] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa08f9389b84169892b6ff041ff687d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "555c0231e4aa41b497efbd807a156f96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43133 } } peers { permanent_uuid: "b1616d2721664fddb9596c61d9ee3e9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42969 } }
I20260620 14:50:04.229310 18387 raft_consensus.cc:374] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:04.229353 18387 raft_consensus.cc:729] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1616d2721664fddb9596c61d9ee3e9c, State: Initialized, Role: FOLLOWER
I20260620 14:50:04.229485 18388 ts_tablet_manager.cc:1428] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:50:04.229470 18387 consensus_queue.cc:260] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa08f9389b84169892b6ff041ff687d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "555c0231e4aa41b497efbd807a156f96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43133 } } peers { permanent_uuid: "b1616d2721664fddb9596c61d9ee3e9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42969 } }
I20260620 14:50:04.229625 18111 heartbeater.cc:503] Master 127.0.98.62:45019 was elected leader, sending a full tablet report...
I20260620 14:50:04.229686 18387 ts_tablet_manager.cc:1428] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:04.229799 18242 heartbeater.cc:503] Master 127.0.98.62:45019 was elected leader, sending a full tablet report...
I20260620 14:50:04.230252 18391 tablet_bootstrap.cc:492] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96: No bootstrap required, opened a new log
I20260620 14:50:04.230307 18391 ts_tablet_manager.cc:1397] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:50:04.232169 18391 raft_consensus.cc:348] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa08f9389b84169892b6ff041ff687d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "555c0231e4aa41b497efbd807a156f96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43133 } } peers { permanent_uuid: "b1616d2721664fddb9596c61d9ee3e9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42969 } }
I20260620 14:50:04.232288 18391 raft_consensus.cc:374] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:04.232308 18391 raft_consensus.cc:729] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555c0231e4aa41b497efbd807a156f96, State: Initialized, Role: FOLLOWER
I20260620 14:50:04.232385 18391 consensus_queue.cc:260] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa08f9389b84169892b6ff041ff687d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "555c0231e4aa41b497efbd807a156f96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43133 } } peers { permanent_uuid: "b1616d2721664fddb9596c61d9ee3e9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42969 } }
I20260620 14:50:04.232589 18391 ts_tablet_manager.cc:1428] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:50:04.232649 18373 heartbeater.cc:503] Master 127.0.98.62:45019 was elected leader, sending a full tablet report...
W20260620 14:50:04.340404 18243 tablet.cc:2369] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:50:04.376004 18393 raft_consensus.cc:482] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:50:04.376116 18393 raft_consensus.cc:504] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa08f9389b84169892b6ff041ff687d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "555c0231e4aa41b497efbd807a156f96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43133 } } peers { permanent_uuid: "b1616d2721664fddb9596c61d9ee3e9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42969 } }
I20260620 14:50:04.376478 18393 leader_election.cc:288] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 555c0231e4aa41b497efbd807a156f96 (127.0.98.3:43133), b1616d2721664fddb9596c61d9ee3e9c (127.0.98.2:42969)
I20260620 14:50:04.380220 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:04.376741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44825 (local address 127.0.98.2:42969)
0620 14:50:04.376790 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:04.376791 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:04.376801 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:50:04.376880 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:04.376881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:04.376891 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:04.376952 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:04.377097 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.377837 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.379300 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.379877 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.379945 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:04.379955 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:04.380111 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 14:50:04.380149 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:50:04.380220 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:04.376765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45279 (local address 127.0.98.3:43133)
0620 14:50:04.376816 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:04.376818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:04.376941 (+ 123us) server_negotiation.cc:407] Connection header received
0620 14:50:04.376972 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:04.376973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:04.376981 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:04.377033 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:04.377144 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.377834 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.379300 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.379884 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.379945 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:04.379955 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:04.380111 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 14:50:04.380149 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:50:04.380256 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:04.376661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42969 (local address 127.0.98.1:44825)
0620 14:50:04.376713 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:04.376725 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:04.376789 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:04.376961 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:04.376961 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:04.376963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:04.377032 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:04.377034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.377902 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.377902 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:04.379211 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:04.379213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.379949 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.379950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:04.380039 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:04.380043 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:50:04.380101 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:50:04.380145 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":54}
I20260620 14:50:04.380261 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:04.376670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43133 (local address 127.0.98.1:45279)
0620 14:50:04.376876 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:04.376885 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:04.376929 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:04.377036 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:04.377036 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:04.377037 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:04.377089 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:04.377090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.377905 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.377905 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:04.379208 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:04.379212 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.379950 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.379951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:04.380045 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:04.380047 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:50:04.380101 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:50:04.380144 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":101,"thread_start_us":89,"threads_started":1}
I20260620 14:50:04.380615 18196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "375b88c14cf74ee2becb8a2c2eac1bc8" candidate_uuid: "2fa08f9389b84169892b6ff041ff687d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1616d2721664fddb9596c61d9ee3e9c" is_pre_election: true
I20260620 14:50:04.380646 18327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "375b88c14cf74ee2becb8a2c2eac1bc8" candidate_uuid: "2fa08f9389b84169892b6ff041ff687d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555c0231e4aa41b497efbd807a156f96" is_pre_election: true
I20260620 14:50:04.380786 18196 raft_consensus.cc:2454] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fa08f9389b84169892b6ff041ff687d in term 0.
I20260620 14:50:04.380786 18327 raft_consensus.cc:2454] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fa08f9389b84169892b6ff041ff687d in term 0.
I20260620 14:50:04.380995 17999 leader_election.cc:302] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa08f9389b84169892b6ff041ff687d, b1616d2721664fddb9596c61d9ee3e9c; no voters:
I20260620 14:50:04.381134 18393 raft_consensus.cc:2781] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:50:04.381193 18393 raft_consensus.cc:482] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:50:04.381215 18393 raft_consensus.cc:3037] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:04.381883 18393 raft_consensus.cc:504] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa08f9389b84169892b6ff041ff687d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "555c0231e4aa41b497efbd807a156f96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43133 } } peers { permanent_uuid: "b1616d2721664fddb9596c61d9ee3e9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42969 } }
I20260620 14:50:04.382001 18393 leader_election.cc:288] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d [CANDIDATE]: Term 1 election: Requested vote from peers 555c0231e4aa41b497efbd807a156f96 (127.0.98.3:43133), b1616d2721664fddb9596c61d9ee3e9c (127.0.98.2:42969)
I20260620 14:50:04.382148 18327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "375b88c14cf74ee2becb8a2c2eac1bc8" candidate_uuid: "2fa08f9389b84169892b6ff041ff687d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555c0231e4aa41b497efbd807a156f96"
I20260620 14:50:04.382159 18196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "375b88c14cf74ee2becb8a2c2eac1bc8" candidate_uuid: "2fa08f9389b84169892b6ff041ff687d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1616d2721664fddb9596c61d9ee3e9c"
I20260620 14:50:04.382233 18327 raft_consensus.cc:3037] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:04.382257 18196 raft_consensus.cc:3037] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:04.382910 18327 raft_consensus.cc:2454] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa08f9389b84169892b6ff041ff687d in term 1.
I20260620 14:50:04.383049 18196 raft_consensus.cc:2454] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa08f9389b84169892b6ff041ff687d in term 1.
I20260620 14:50:04.383071 17998 leader_election.cc:302] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa08f9389b84169892b6ff041ff687d, 555c0231e4aa41b497efbd807a156f96; no voters:
I20260620 14:50:04.383251 18393 raft_consensus.cc:2781] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:04.383397 18393 raft_consensus.cc:686] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d [term 1 LEADER]: Becoming Leader. State: Replica: 2fa08f9389b84169892b6ff041ff687d, State: Running, Role: LEADER
I20260620 14:50:04.383522 18393 consensus_queue.cc:237] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa08f9389b84169892b6ff041ff687d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "555c0231e4aa41b497efbd807a156f96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43133 } } peers { permanent_uuid: "b1616d2721664fddb9596c61d9ee3e9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42969 } }
I20260620 14:50:04.384347 17922 catalog_manager.cc:5495] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fa08f9389b84169892b6ff041ff687d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2fa08f9389b84169892b6ff041ff687d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa08f9389b84169892b6ff041ff687d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "555c0231e4aa41b497efbd807a156f96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1616d2721664fddb9596c61d9ee3e9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42969 } health_report { overall_health: UNKNOWN } } }
I20260620 14:50:04.402384 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:04.400082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52084 (local address 127.0.98.1:32863)
0620 14:50:04.400167 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:04.400169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:04.400177 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:50:04.400211 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:04.400212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:04.400231 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:04.400297 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:04.400388 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.400850 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.401584 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:04.401780 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:04.401843 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:04.402050 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:04.402168 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:04.402250 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:04.402264 (+ 14us) server_negotiation.cc:299] Negotiation successful
0620 14:50:04.402285 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:50:04.404302 18327 raft_consensus.cc:1261] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96 [term 1 FOLLOWER]: Refusing update from remote peer 2fa08f9389b84169892b6ff041ff687d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:04.404302 18196 raft_consensus.cc:1261] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c [term 1 FOLLOWER]: Refusing update from remote peer 2fa08f9389b84169892b6ff041ff687d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:04.404670 18397 consensus_queue.cc:1035] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1616d2721664fddb9596c61d9ee3e9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:04.404795 18393 consensus_queue.cc:1035] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 2fa08f9389b84169892b6ff041ff687d [LEADER]: Connected to new peer: Peer: permanent_uuid: "555c0231e4aa41b497efbd807a156f96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:04.407011 18400 mvcc.cc:205] Tried to move back new op lower bound from 7298936850037178368 to 7298936849955463168. Current Snapshot: MvccSnapshot[applied={T|T < 7298936850037178368}]
I20260620 14:50:04.407548 18401 mvcc.cc:205] Tried to move back new op lower bound from 7298936850037178368 to 7298936849955463168. Current Snapshot: MvccSnapshot[applied={T|T < 7298936850037178368}]
I20260620 14:50:04.409349 17923 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:04.409430 17923 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:04.410107 17923 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe4ab17c12a546dc90e3bdc460e5ca9e: Sending DeleteTablet for 3 replicas of tablet 375b88c14cf74ee2becb8a2c2eac1bc8
I20260620 14:50:04.410435 18176 tablet_service.cc:1514] Processing DeleteTablet for tablet 375b88c14cf74ee2becb8a2c2eac1bc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:04 UTC) from {username='slave'} at 127.0.0.1:51980
I20260620 14:50:04.410461 18307 tablet_service.cc:1514] Processing DeleteTablet for tablet 375b88c14cf74ee2becb8a2c2eac1bc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:04 UTC) from {username='slave'} at 127.0.0.1:39884
I20260620 14:50:04.410450 18045 tablet_service.cc:1514] Processing DeleteTablet for tablet 375b88c14cf74ee2becb8a2c2eac1bc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:04 UTC) from {username='slave'} at 127.0.0.1:52078
I20260620 14:50:04.410727 18412 tablet_replica.cc:330] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c: stopping tablet replica
I20260620 14:50:04.410820 18412 raft_consensus.cc:2229] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:04.410918 18412 raft_consensus.cc:2258] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:04.410928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 17983
I20260620 14:50:04.412482 18411 tablet_replica.cc:330] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96: stopping tablet replica
I20260620 14:50:04.412566 18411 raft_consensus.cc:2229] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:04.412662 18411 raft_consensus.cc:2258] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:04.413056 18412 ts_tablet_manager.cc:1905] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:04.413086 18411 ts_tablet_manager.cc:1905] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:04.414453 18411 ts_tablet_manager.cc:1918] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:04.414520 18411 log.cc:1196] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/375b88c14cf74ee2becb8a2c2eac1bc8
I20260620 14:50:04.414608 18412 ts_tablet_manager.cc:1918] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:04.414710 18412 log.cc:1196] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/375b88c14cf74ee2becb8a2c2eac1bc8
I20260620 14:50:04.414762 18411 ts_tablet_manager.cc:1939] T 375b88c14cf74ee2becb8a2c2eac1bc8 P 555c0231e4aa41b497efbd807a156f96: Deleting consensus metadata
I20260620 14:50:04.414979 18412 ts_tablet_manager.cc:1939] T 375b88c14cf74ee2becb8a2c2eac1bc8 P b1616d2721664fddb9596c61d9ee3e9c: Deleting consensus metadata
I20260620 14:50:04.415136 17907 catalog_manager.cc:4841] TS 555c0231e4aa41b497efbd807a156f96 (127.0.98.3:43133): tablet 375b88c14cf74ee2becb8a2c2eac1bc8 (table test-table [id=caabed09bb1342ce8cff6e3fab855823]) successfully deleted
I20260620 14:50:04.415355 17908 catalog_manager.cc:4841] TS b1616d2721664fddb9596c61d9ee3e9c (127.0.98.2:42969): tablet 375b88c14cf74ee2becb8a2c2eac1bc8 (table test-table [id=caabed09bb1342ce8cff6e3fab855823]) successfully deleted
W20260620 14:50:04.416576 17909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32863 (error 108)
I20260620 14:50:04.417034 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:04.416927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32863 (local address 127.0.0.1:52090)
0620 14:50:04.416969 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:04.417007 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 14:50:04.417169 17909 catalog_manager.cc:4584] TS 2fa08f9389b84169892b6ff041ff687d (127.0.98.1:32863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 375b88c14cf74ee2becb8a2c2eac1bc8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32863: connect: Connection refused (error 111)
I20260620 14:50:04.416715 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 18114
I20260620 14:50:04.421895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 18245
I20260620 14:50:04.425524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 17892
2026-06-20T14:50:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (762 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:04 dist-test-slave-cx89 krb5kdc[18416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:04 dist-test-slave-cx89 krb5kdc[18416](info): set up 2 sockets
Jun 20 14:50:04 dist-test-slave-cx89 krb5kdc[18416](info): commencing operation
krb5kdc: starting...
W20260620 14:50:05.391142 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:05 dist-test-slave-cx89 krb5kdc[18416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:50:05.499110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37615
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:05.576860 18429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:05.577071 18429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:05.577093 18429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:05.578222 18429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:05.578261 18429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:05.578274 18429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:05.578285 18429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:05.578295 18429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:05.579771 18429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:05.579986 18429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:05.580173 18429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:05.584839 18435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:05.584858 18434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:05.584887 18429 server_base.cc:1034] running on GCE node
W20260620 14:50:05.584834 18437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:05.585294 18429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:05.585579 18429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:05.586735 18429 hybrid_clock.cc:648] HybridClock initialized: now 1781967005586721 us; error 27 us; skew 500 ppm
Jun 20 14:50:05 dist-test-slave-cx89 krb5kdc[18416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:05.589010 18429 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:50:05.589440 18429 webserver.cc:460] Webserver started at http://127.0.98.62:42127/ using document root <none> and password file <none>
I20260620 14:50:05.589653 18429 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:05.589700 18429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:05.589819 18429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:05.590834 18429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "b15f92f5bf004f4997f34b67dc43b674"
format_stamp: "Formatted at 2026-06-20 14:50:05 on dist-test-slave-cx89"
server_key: "a97f4e2b161f81a74a00a680aa9a3f3d"
server_key_iv: "d939c737e4e077059f71e06395f151c4"
server_key_version: "encryptionkey@0"
I20260620 14:50:05.591149 18429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "b15f92f5bf004f4997f34b67dc43b674"
format_stamp: "Formatted at 2026-06-20 14:50:05 on dist-test-slave-cx89"
server_key: "a97f4e2b161f81a74a00a680aa9a3f3d"
server_key_iv: "d939c737e4e077059f71e06395f151c4"
server_key_version: "encryptionkey@0"
I20260620 14:50:05.592408 18429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:50:05.593060 18444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:05.593199 18429 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:50:05.593276 18429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "b15f92f5bf004f4997f34b67dc43b674"
format_stamp: "Formatted at 2026-06-20 14:50:05 on dist-test-slave-cx89"
server_key: "a97f4e2b161f81a74a00a680aa9a3f3d"
server_key_iv: "d939c737e4e077059f71e06395f151c4"
server_key_version: "encryptionkey@0"
I20260620 14:50:05.593326 18429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:05.634249 18429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:05.636248 18429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:05.636358 18429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:05.639976 18429 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37615
I20260620 14:50:05.640024 18496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37615 every 8 connection(s)
I20260620 14:50:05.640345 18429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:05.641137 18497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:05.643395 18497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674: Bootstrap starting.
I20260620 14:50:05.643743 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 18429
I20260620 14:50:05.643839 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:05.644002 18497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:05.644012 392 external_mini_cluster.cc:1428] Setting key 835564013c35ab8d602a8caa80b01517
I20260620 14:50:05.644259 18497 log.cc:824] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:05.644997 18497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674: No bootstrap required, opened a new log
I20260620 14:50:05.647212 18497 raft_consensus.cc:348] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15f92f5bf004f4997f34b67dc43b674" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37615 } }
I20260620 14:50:05.647378 18497 raft_consensus.cc:374] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:05.647410 18497 raft_consensus.cc:729] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15f92f5bf004f4997f34b67dc43b674, State: Initialized, Role: FOLLOWER
I20260620 14:50:05.647569 18497 consensus_queue.cc:260] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15f92f5bf004f4997f34b67dc43b674" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37615 } }
I20260620 14:50:05.647655 18497 raft_consensus.cc:388] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:05.647684 18497 raft_consensus.cc:482] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:05.647728 18497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:05.648255 18497 raft_consensus.cc:504] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15f92f5bf004f4997f34b67dc43b674" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37615 } }
I20260620 14:50:05.648360 18497 leader_election.cc:302] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b15f92f5bf004f4997f34b67dc43b674; no voters:
I20260620 14:50:05.648515 18497 leader_election.cc:288] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:05.648561 18502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:05.648702 18502 raft_consensus.cc:686] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674 [term 1 LEADER]: Becoming Leader. State: Replica: b15f92f5bf004f4997f34b67dc43b674, State: Running, Role: LEADER
I20260620 14:50:05.648767 18497 sys_catalog.cc:564] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:05.648790 18502 consensus_queue.cc:237] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15f92f5bf004f4997f34b67dc43b674" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37615 } }
I20260620 14:50:05.649156 18504 sys_catalog.cc:455] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b15f92f5bf004f4997f34b67dc43b674. Latest consensus state: current_term: 1 leader_uuid: "b15f92f5bf004f4997f34b67dc43b674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15f92f5bf004f4997f34b67dc43b674" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37615 } } }
I20260620 14:50:05.649238 18504 sys_catalog.cc:458] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:05.649163 18503 sys_catalog.cc:455] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b15f92f5bf004f4997f34b67dc43b674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15f92f5bf004f4997f34b67dc43b674" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37615 } } }
I20260620 14:50:05.649405 18503 sys_catalog.cc:458] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:05.649511 18509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:05.649968 18509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 14:50:05 dist-test-slave-cx89 krb5kdc[18416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:50:05.654397 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:05.644443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43014 (local address 127.0.98.62:37615)
0620 14:50:05.644609 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:05.644619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:05.644651 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:50:05.645219 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:05.645229 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:05.645291 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:05.645442 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:05.646025 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:05.647391 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:05.648216 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:05.648824 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:05.649275 (+ 451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:05.651587 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:05.651591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:05.651592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:05.651603 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:05.653049 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:05.653358 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:05.653361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:05.653361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:05.653448 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:05.653613 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:05.653613 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:05.653614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:05.653842 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:05.653977 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:05.654096 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 14:50:05.654237 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260620 14:50:05.655498 18509 catalog_manager.cc:1347] Generated new cluster ID: 04f237d59d144845a7efbdb12ab39176
I20260620 14:50:05.655547 18509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:05.666543 18509 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:05.667268 18509 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:05.672928 18509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674: Generated new TSK 0
I20260620 14:50:05.673112 18509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:50:05.700469 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37615
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:05.779359 18523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:05.779597 18523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:05.779620 18523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:05.780786 18523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:05.780826 18523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:05.780853 18523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:50:05.782342 18523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:05.782553 18523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:05.782765 18523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:05.787566 18528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:05.787606 18529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:05.787593 18523 server_base.cc:1034] running on GCE node
W20260620 14:50:05.787564 18531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:05.787936 18523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:05.788158 18523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:05.789288 18523 hybrid_clock.cc:648] HybridClock initialized: now 1781967005789277 us; error 29 us; skew 500 ppm
Jun 20 14:50:05 dist-test-slave-cx89 krb5kdc[18416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:05.791600 18523 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:50:05.792021 18523 webserver.cc:460] Webserver started at http://127.0.98.1:46091/ using document root <none> and password file <none>
I20260620 14:50:05.792198 18523 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:05.792235 18523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:05.792325 18523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:05.793274 18523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "9c1e3f9d265b46be984f9f9eebda6c38"
format_stamp: "Formatted at 2026-06-20 14:50:05 on dist-test-slave-cx89"
server_key: "e0bf347c39cb93f4f2e2aa0ecffa6a01"
server_key_iv: "d19ea09bc4c0e3c1ed9f6442d9e8ed62"
server_key_version: "encryptionkey@0"
I20260620 14:50:05.793542 18523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9c1e3f9d265b46be984f9f9eebda6c38"
format_stamp: "Formatted at 2026-06-20 14:50:05 on dist-test-slave-cx89"
server_key: "e0bf347c39cb93f4f2e2aa0ecffa6a01"
server_key_iv: "d19ea09bc4c0e3c1ed9f6442d9e8ed62"
server_key_version: "encryptionkey@0"
I20260620 14:50:05.794662 18523 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:50:05.795356 18538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:05.795528 18523 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:50:05.795611 18523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "9c1e3f9d265b46be984f9f9eebda6c38"
format_stamp: "Formatted at 2026-06-20 14:50:05 on dist-test-slave-cx89"
server_key: "e0bf347c39cb93f4f2e2aa0ecffa6a01"
server_key_iv: "d19ea09bc4c0e3c1ed9f6442d9e8ed62"
server_key_version: "encryptionkey@0"
I20260620 14:50:05.795678 18523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:05.805699 18523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:05.807685 18523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:05.807804 18523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:05.807996 18523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:05.808282 18523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:05.808312 18523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:05.808341 18523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:05.808365 18523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:05.813773 18523 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35185
I20260620 14:50:05.813875 18651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35185 every 8 connection(s)
I20260620 14:50:05.814175 18523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:50:05.814651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 18523
I20260620 14:50:05.814733 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:50:05.814929 392 external_mini_cluster.cc:1428] Setting key ca951e5613e1b9ded8c88024e5d0402b
Jun 20 14:50:05 dist-test-slave-cx89 krb5kdc[18416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:50:05.825465 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:05.815529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58029 (local address 127.0.98.62:37615)
0620 14:50:05.815714 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:05.815716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:05.816662 (+ 946us) server_negotiation.cc:407] Connection header received
0620 14:50:05.817846 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:05.817848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:05.817861 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:05.817935 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:05.818544 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:05.819284 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:05.820318 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:05.820631 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:05.820701 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:05.822864 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:05.822865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:05.822866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:05.822874 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:05.824138 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:05.824481 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:05.824481 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:05.824482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:05.824524 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:05.824777 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:05.824777 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:05.824777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:05.824865 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:05.824909 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:05.825279 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 14:50:05.825368 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260620 14:50:05.825538 18654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:05.815550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37615 (local address 127.0.98.1:58029)
0620 14:50:05.816501 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:05.816569 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:05.817723 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:05.817993 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:05.818003 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:05.818261 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:05.818455 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:05.818460 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:05.819379 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:05.819380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:05.820231 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:05.820234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:05.820706 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:05.820706 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:05.820798 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:05.821256 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:05.821265 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:05.822771 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:05.824207 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:05.824213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:05.824215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:05.824421 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:05.824597 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:05.824598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:05.824598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:05.824730 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:05.824933 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:05.824939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:05.825109 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 14:50:05.825213 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:50:05.825362 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"thread_start_us":83,"threads_started":1}
I20260620 14:50:05.826002 18652 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37615
I20260620 14:50:05.826098 18652 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:05.826287 18652 heartbeater.cc:511] Master 127.0.98.62:37615 requested a full tablet report, sending...
I20260620 14:50:05.826814 18461 ts_manager.cc:194] Registered new tserver with Master: 9c1e3f9d265b46be984f9f9eebda6c38 (127.0.98.1:35185)
I20260620 14:50:05.827476 18461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:50:05.838169 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37615
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:05.915309 18657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:05.915529 18657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:05.915562 18657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:05.916709 18657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:05.916750 18657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:05.916783 18657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:50:05.918226 18657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:05.918450 18657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:05.918653 18657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:05.923261 18665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:05.923364 18663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:05.923455 18657 server_base.cc:1034] running on GCE node
W20260620 14:50:05.923243 18662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:05.923738 18657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:05.923986 18657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:05.925164 18657 hybrid_clock.cc:648] HybridClock initialized: now 1781967005925150 us; error 28 us; skew 500 ppm
Jun 20 14:50:05 dist-test-slave-cx89 krb5kdc[18416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:05.927341 18657 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:50:05.927748 18657 webserver.cc:460] Webserver started at http://127.0.98.2:45081/ using document root <none> and password file <none>
I20260620 14:50:05.927933 18657 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:05.927979 18657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:05.928103 18657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:05.929096 18657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "ded4fb0065fe49338987112908fb6513"
format_stamp: "Formatted at 2026-06-20 14:50:05 on dist-test-slave-cx89"
server_key: "b0efe33e167a09f3ca098bd4ecd1e2be"
server_key_iv: "c00b80d915a6ce686619e91780563c08"
server_key_version: "encryptionkey@0"
I20260620 14:50:05.929410 18657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ded4fb0065fe49338987112908fb6513"
format_stamp: "Formatted at 2026-06-20 14:50:05 on dist-test-slave-cx89"
server_key: "b0efe33e167a09f3ca098bd4ecd1e2be"
server_key_iv: "c00b80d915a6ce686619e91780563c08"
server_key_version: "encryptionkey@0"
I20260620 14:50:05.930552 18657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:50:05.931277 18672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:05.931469 18657 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:05.931557 18657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "ded4fb0065fe49338987112908fb6513"
format_stamp: "Formatted at 2026-06-20 14:50:05 on dist-test-slave-cx89"
server_key: "b0efe33e167a09f3ca098bd4ecd1e2be"
server_key_iv: "c00b80d915a6ce686619e91780563c08"
server_key_version: "encryptionkey@0"
I20260620 14:50:05.931624 18657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:05.947847 18657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:05.949786 18657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:05.949918 18657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:05.950130 18657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:05.950449 18657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:05.950484 18657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:05.950512 18657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:05.950536 18657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:05.955991 18657 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44865
I20260620 14:50:05.956043 18785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44865 every 8 connection(s)
I20260620 14:50:05.956331 18657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:50:05 dist-test-slave-cx89 krb5kdc[18416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:50:05.962477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 18657
I20260620 14:50:05.962585 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:50:05.962774 392 external_mini_cluster.cc:1428] Setting key 9ac5c9143c5023d9e023a1fec6fbc894
I20260620 14:50:05.965397 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:05.956987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33967 (local address 127.0.98.62:37615)
0620 14:50:05.957041 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:05.957043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:05.957235 (+ 192us) server_negotiation.cc:407] Connection header received
0620 14:50:05.958139 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:05.958140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:05.958153 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:05.958205 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:05.958801 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:05.959285 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:05.960314 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:05.960504 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:05.960562 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:05.962641 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:05.962643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:05.962643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:05.962652 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:05.964025 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:05.964468 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:05.964469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:05.964469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:05.964520 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:05.964784 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:05.964785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:05.964786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:05.964920 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:05.964987 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:05.965224 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 14:50:05.965307 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:50:05.965477 18788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:05.956971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37615 (local address 127.0.98.2:33967)
0620 14:50:05.957113 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:05.957155 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:05.958034 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:05.958277 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:05.958280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:05.958521 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:05.958713 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:05.958717 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:05.959380 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:05.959381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:05.960230 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:05.960233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:05.960571 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:05.960573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:05.960691 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:05.961149 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:05.961157 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:05.962526 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:05.964102 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:05.964110 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:05.964113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:05.964398 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:05.964581 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:05.964582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:05.964582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:05.964716 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:05.964979 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:05.964983 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:05.965111 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 14:50:05.965194 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:50:05.965313 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260620 14:50:05.965912 18786 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37615
I20260620 14:50:05.966006 18786 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:05.966190 18786 heartbeater.cc:511] Master 127.0.98.62:37615 requested a full tablet report, sending...
I20260620 14:50:05.966579 18461 ts_manager.cc:194] Registered new tserver with Master: ded4fb0065fe49338987112908fb6513 (127.0.98.2:44865)
I20260620 14:50:05.966948 18461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33967
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:50:05.985697 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37615
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:06.064108 18791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:06.064312 18791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:06.064334 18791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:06.065493 18791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:06.065537 18791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:06.065563 18791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:50:06.067008 18791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:06.067225 18791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:06.067443 18791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:06.071981 18799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:06.072019 18797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:06.072077 18791 server_base.cc:1034] running on GCE node
W20260620 14:50:06.072120 18796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:06.072395 18791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:06.072641 18791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:06.073778 18791 hybrid_clock.cc:648] HybridClock initialized: now 1781967006073764 us; error 26 us; skew 500 ppm
Jun 20 14:50:06 dist-test-slave-cx89 krb5kdc[18416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:06.076117 18791 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:50:06.076503 18791 webserver.cc:460] Webserver started at http://127.0.98.3:43105/ using document root <none> and password file <none>
I20260620 14:50:06.076706 18791 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:06.076753 18791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:06.076871 18791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:06.077873 18791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "f5858427b8aa48b9b54e053523d23cad"
format_stamp: "Formatted at 2026-06-20 14:50:06 on dist-test-slave-cx89"
server_key: "b48b6742e83b56565a78e5fc58ce039f"
server_key_iv: "50b46ea42a3f0469e2140809833525c5"
server_key_version: "encryptionkey@0"
I20260620 14:50:06.078188 18791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f5858427b8aa48b9b54e053523d23cad"
format_stamp: "Formatted at 2026-06-20 14:50:06 on dist-test-slave-cx89"
server_key: "b48b6742e83b56565a78e5fc58ce039f"
server_key_iv: "50b46ea42a3f0469e2140809833525c5"
server_key_version: "encryptionkey@0"
I20260620 14:50:06.079361 18791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:50:06.080034 18806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:06.080188 18791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:06.080266 18791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "f5858427b8aa48b9b54e053523d23cad"
format_stamp: "Formatted at 2026-06-20 14:50:06 on dist-test-slave-cx89"
server_key: "b48b6742e83b56565a78e5fc58ce039f"
server_key_iv: "50b46ea42a3f0469e2140809833525c5"
server_key_version: "encryptionkey@0"
I20260620 14:50:06.080325 18791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:06.102770 18791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:06.104879 18791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:06.105016 18791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:06.105232 18791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:06.105515 18791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:06.105545 18791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:06.105574 18791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:06.105597 18791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:06.111073 18791 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32873
I20260620 14:50:06.111114 18919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32873 every 8 connection(s)
I20260620 14:50:06.111407 18791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:50:06 dist-test-slave-cx89 krb5kdc[18416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:50:06.120137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 18791
I20260620 14:50:06.120252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:50:06.120285 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.112113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49137 (local address 127.0.98.62:37615)
0620 14:50:06.112161 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:06.112163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:06.112351 (+ 188us) server_negotiation.cc:407] Connection header received
0620 14:50:06.113240 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:06.113241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:06.113251 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:06.113311 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:06.113899 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.114396 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.115435 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.115649 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.115713 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.117751 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:06.117752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:06.117753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:06.117759 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:06.118868 (+ 1109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:06.119262 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:06.119263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:06.119263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:06.119306 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:06.119599 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:06.119601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:06.119601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:06.119749 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:06.119809 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:06.120095 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 14:50:06.120192 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:50:06.120437 392 external_mini_cluster.cc:1428] Setting key 9ea14d68c2117c7c7052cfd672e429b5
I20260620 14:50:06.120483 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.112078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37615 (local address 127.0.98.3:49137)
0620 14:50:06.112228 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:06.112270 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:06.113152 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:06.113366 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:06.113370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:06.113615 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:06.113813 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.113818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.114480 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.114481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.115308 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.115310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.115723 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.115724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.115834 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.116326 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:06.116334 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:06.117660 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:06.118953 (+ 1293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:06.118959 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:06.118961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:06.119199 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:06.119353 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:06.119354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:06.119354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:06.119504 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:06.119826 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:06.119831 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:06.119956 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 14:50:06.120055 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:50:06.120210 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260620 14:50:06.120934 18920 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37615
I20260620 14:50:06.121039 18920 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:06.121244 18920 heartbeater.cc:511] Master 127.0.98.62:37615 requested a full tablet report, sending...
I20260620 14:50:06.121614 18460 ts_manager.cc:194] Registered new tserver with Master: f5858427b8aa48b9b54e053523d23cad (127.0.98.3:32873)
I20260620 14:50:06.121953 18460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49137
I20260620 14:50:06.132084 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:50:06.139353 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.133401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.0.98.62:37615)
0620 14:50:06.133443 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:06.133445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:06.133566 (+ 121us) server_negotiation.cc:407] Connection header received
0620 14:50:06.133620 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:06.133621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:06.133631 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:06.133678 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:06.134277 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.134756 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.135526 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.135711 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.135762 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.136998 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:06.136999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:06.137000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:06.137007 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:06.138230 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:06.138593 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:06.138594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:06.138594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:06.138645 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:06.138858 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:06.138859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:06.138859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:06.138992 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:06.139092 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:06.139204 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 14:50:06.139262 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:50:06.140977 18460 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:50:06.142545 18460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:50:06.148626 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.144568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44865 (local address 127.0.0.1:51208)
0620 14:50:06.145278 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:06.145288 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:06.145338 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:06.145640 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:06.145640 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:06.145642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:06.145673 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.145674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.146709 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.146710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.147741 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.147743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.148349 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.148350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.148425 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.148438 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:50:06.148497 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:50:06.148550 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":51,"threads_started":1}
I20260620 14:50:06.148690 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.144734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51208 (local address 127.0.98.2:44865)
0620 14:50:06.145156 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:06.145160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:06.145394 (+ 234us) server_negotiation.cc:407] Connection header received
0620 14:50:06.145435 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:06.145437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:06.145451 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:06.145513 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:06.145751 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.146621 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.147807 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.148278 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.148339 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.148398 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:06.148516 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 14:50:06.148562 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":68,"threads_started":1}
I20260620 14:50:06.148758 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.144557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32873 (local address 127.0.0.1:58678)
0620 14:50:06.144730 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:06.144770 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:06.144833 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:06.145051 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:06.145054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:06.145057 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:06.145139 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.145141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.146359 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.146360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.147569 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.147572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.148496 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.148497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.148552 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.148555 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:50:06.148640 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:50:06.148679 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":1,"spinlock_wait_cycles":768,"thread_start_us":92,"threads_started":1}
I20260620 14:50:06.149060 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.144732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58678 (local address 127.0.98.3:32873)
0620 14:50:06.144905 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:06.144910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:06.144920 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:50:06.144956 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:06.144958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:06.144974 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:06.145042 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:06.145229 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.146259 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.147634 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.148350 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.148416 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.148464 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:06.148921 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 14:50:06.148967 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":73,"threads_started":1}
I20260620 14:50:06.149315 18720 tablet_service.cc:1467] Processing CreateTablet for tablet b3309690fb3843b69ca745dadc3ecc69 (DEFAULT_TABLE table=test-table [id=18b0e9fc3c284649ac7aed089c9a620b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:06.149641 18720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3309690fb3843b69ca745dadc3ecc69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:06.150084 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.144557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35185 (local address 127.0.0.1:50218)
0620 14:50:06.145125 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:06.145135 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:06.145193 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:06.145434 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:06.145434 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:06.145436 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:06.145479 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.145480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.147457 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.147458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.148533 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.148534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.149868 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.149869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.149930 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.149933 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:50:06.149978 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 14:50:06.150016 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260620 14:50:06.150106 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.144816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50218 (local address 127.0.98.1:35185)
0620 14:50:06.144984 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:06.144989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:06.145216 (+ 227us) server_negotiation.cc:407] Connection header received
0620 14:50:06.145261 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:06.145263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:06.145278 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:06.145351 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:06.145549 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.146269 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.149032 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.149715 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.149761 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.149816 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:06.149991 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:50:06.150029 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260620 14:50:06.150686 18586 tablet_service.cc:1467] Processing CreateTablet for tablet b3309690fb3843b69ca745dadc3ecc69 (DEFAULT_TABLE table=test-table [id=18b0e9fc3c284649ac7aed089c9a620b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:06.150946 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3309690fb3843b69ca745dadc3ecc69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:06.152307 18934 tablet_bootstrap.cc:492] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513: Bootstrap starting.
I20260620 14:50:06.152839 18853 tablet_service.cc:1467] Processing CreateTablet for tablet b3309690fb3843b69ca745dadc3ecc69 (DEFAULT_TABLE table=test-table [id=18b0e9fc3c284649ac7aed089c9a620b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:06.153091 18934 tablet_bootstrap.cc:654] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:06.153152 18853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3309690fb3843b69ca745dadc3ecc69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:06.153556 18935 tablet_bootstrap.cc:492] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38: Bootstrap starting.
I20260620 14:50:06.153611 18934 log.cc:824] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:06.154140 18935 tablet_bootstrap.cc:654] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:06.154363 18934 tablet_bootstrap.cc:492] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513: No bootstrap required, opened a new log
I20260620 14:50:06.154426 18934 ts_tablet_manager.cc:1397] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:50:06.154665 18935 log.cc:824] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:06.155242 18935 tablet_bootstrap.cc:492] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38: No bootstrap required, opened a new log
I20260620 14:50:06.155298 18935 ts_tablet_manager.cc:1397] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:50:06.156333 18934 raft_consensus.cc:348] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5858427b8aa48b9b54e053523d23cad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32873 } } peers { permanent_uuid: "ded4fb0065fe49338987112908fb6513" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44865 } } peers { permanent_uuid: "9c1e3f9d265b46be984f9f9eebda6c38" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } }
I20260620 14:50:06.156471 18934 raft_consensus.cc:374] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:06.156492 18934 raft_consensus.cc:729] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded4fb0065fe49338987112908fb6513, State: Initialized, Role: FOLLOWER
I20260620 14:50:06.156584 18934 consensus_queue.cc:260] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5858427b8aa48b9b54e053523d23cad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32873 } } peers { permanent_uuid: "ded4fb0065fe49338987112908fb6513" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44865 } } peers { permanent_uuid: "9c1e3f9d265b46be984f9f9eebda6c38" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } }
I20260620 14:50:06.156822 18934 ts_tablet_manager.cc:1428] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:50:06.156853 18786 heartbeater.cc:503] Master 127.0.98.62:37615 was elected leader, sending a full tablet report...
I20260620 14:50:06.156991 18938 tablet_bootstrap.cc:492] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad: Bootstrap starting.
I20260620 14:50:06.157152 18935 raft_consensus.cc:348] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5858427b8aa48b9b54e053523d23cad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32873 } } peers { permanent_uuid: "ded4fb0065fe49338987112908fb6513" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44865 } } peers { permanent_uuid: "9c1e3f9d265b46be984f9f9eebda6c38" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } }
I20260620 14:50:06.157310 18935 raft_consensus.cc:374] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:06.157349 18935 raft_consensus.cc:729] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1e3f9d265b46be984f9f9eebda6c38, State: Initialized, Role: FOLLOWER
I20260620 14:50:06.157454 18935 consensus_queue.cc:260] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5858427b8aa48b9b54e053523d23cad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32873 } } peers { permanent_uuid: "ded4fb0065fe49338987112908fb6513" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44865 } } peers { permanent_uuid: "9c1e3f9d265b46be984f9f9eebda6c38" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } }
I20260620 14:50:06.157574 18938 tablet_bootstrap.cc:654] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:06.157721 18935 ts_tablet_manager.cc:1428] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:50:06.157835 18652 heartbeater.cc:503] Master 127.0.98.62:37615 was elected leader, sending a full tablet report...
I20260620 14:50:06.158094 18938 log.cc:824] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:06.158804 18938 tablet_bootstrap.cc:492] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad: No bootstrap required, opened a new log
I20260620 14:50:06.158862 18938 ts_tablet_manager.cc:1397] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:50:06.160157 18938 raft_consensus.cc:348] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5858427b8aa48b9b54e053523d23cad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32873 } } peers { permanent_uuid: "ded4fb0065fe49338987112908fb6513" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44865 } } peers { permanent_uuid: "9c1e3f9d265b46be984f9f9eebda6c38" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } }
I20260620 14:50:06.160319 18938 raft_consensus.cc:374] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:06.160362 18938 raft_consensus.cc:729] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5858427b8aa48b9b54e053523d23cad, State: Initialized, Role: FOLLOWER
I20260620 14:50:06.160465 18938 consensus_queue.cc:260] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5858427b8aa48b9b54e053523d23cad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32873 } } peers { permanent_uuid: "ded4fb0065fe49338987112908fb6513" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44865 } } peers { permanent_uuid: "9c1e3f9d265b46be984f9f9eebda6c38" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } }
I20260620 14:50:06.160678 18938 ts_tablet_manager.cc:1428] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:50:06.160715 18920 heartbeater.cc:503] Master 127.0.98.62:37615 was elected leader, sending a full tablet report...
W20260620 14:50:06.206868 18787 tablet.cc:2369] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:50:06.248950 18940 raft_consensus.cc:482] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:50:06.249099 18940 raft_consensus.cc:504] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5858427b8aa48b9b54e053523d23cad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32873 } } peers { permanent_uuid: "ded4fb0065fe49338987112908fb6513" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44865 } } peers { permanent_uuid: "9c1e3f9d265b46be984f9f9eebda6c38" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } }
I20260620 14:50:06.249469 18940 leader_election.cc:288] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5858427b8aa48b9b54e053523d23cad (127.0.98.3:32873), ded4fb0065fe49338987112908fb6513 (127.0.98.2:44865)
I20260620 14:50:06.252193 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.249750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58067 (local address 127.0.98.3:32873)
0620 14:50:06.249797 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:06.249800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:06.249809 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:50:06.249908 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:06.249909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:06.249918 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:06.249960 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:06.250137 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.250604 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.251525 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.251881 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.251939 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.251955 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:06.252090 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 14:50:06.252129 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:50:06.252193 18654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.249685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32873 (local address 127.0.98.1:58067)
0620 14:50:06.249734 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:06.249748 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:06.249811 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:06.250012 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:06.250013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:06.250015 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:06.250084 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.250086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.250682 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.250683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.251436 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.251439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.251951 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.251952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.252028 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.252031 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:50:06.252075 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 14:50:06.252117 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 14:50:06.252511 18874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3309690fb3843b69ca745dadc3ecc69" candidate_uuid: "9c1e3f9d265b46be984f9f9eebda6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5858427b8aa48b9b54e053523d23cad" is_pre_election: true
I20260620 14:50:06.252566 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.249840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55073 (local address 127.0.98.2:44865)
0620 14:50:06.249905 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:06.249906 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:06.250033 (+ 127us) server_negotiation.cc:407] Connection header received
0620 14:50:06.250089 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:06.250090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:06.250101 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:06.250167 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:06.250319 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.250805 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.251788 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.252173 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.252243 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.252254 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:06.252453 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 14:50:06.252494 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:50:06.252557 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.249685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44865 (local address 127.0.98.1:55073)
0620 14:50:06.249965 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:06.249975 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:06.250025 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:06.250182 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:06.250183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:06.250186 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:06.250255 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.250257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.250887 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.250888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.251698 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.251701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.252277 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.252278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.252346 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.252350 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:50:06.252432 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:50:06.252471 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":90,"thread_start_us":103,"threads_started":1}
I20260620 14:50:06.252672 18874 raft_consensus.cc:2454] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1e3f9d265b46be984f9f9eebda6c38 in term 0.
I20260620 14:50:06.252863 18740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3309690fb3843b69ca745dadc3ecc69" candidate_uuid: "9c1e3f9d265b46be984f9f9eebda6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded4fb0065fe49338987112908fb6513" is_pre_election: true
I20260620 14:50:06.252925 18541 leader_election.cc:302] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c1e3f9d265b46be984f9f9eebda6c38, f5858427b8aa48b9b54e053523d23cad; no voters:
I20260620 14:50:06.253006 18740 raft_consensus.cc:2454] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1e3f9d265b46be984f9f9eebda6c38 in term 0.
I20260620 14:50:06.253028 18940 raft_consensus.cc:2781] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:50:06.253077 18940 raft_consensus.cc:482] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:50:06.253106 18940 raft_consensus.cc:3037] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:06.253805 18940 raft_consensus.cc:504] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5858427b8aa48b9b54e053523d23cad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32873 } } peers { permanent_uuid: "ded4fb0065fe49338987112908fb6513" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44865 } } peers { permanent_uuid: "9c1e3f9d265b46be984f9f9eebda6c38" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } }
I20260620 14:50:06.253923 18940 leader_election.cc:288] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 [CANDIDATE]: Term 1 election: Requested vote from peers f5858427b8aa48b9b54e053523d23cad (127.0.98.3:32873), ded4fb0065fe49338987112908fb6513 (127.0.98.2:44865)
I20260620 14:50:06.254084 18740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3309690fb3843b69ca745dadc3ecc69" candidate_uuid: "9c1e3f9d265b46be984f9f9eebda6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded4fb0065fe49338987112908fb6513"
I20260620 14:50:06.254110 18874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3309690fb3843b69ca745dadc3ecc69" candidate_uuid: "9c1e3f9d265b46be984f9f9eebda6c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5858427b8aa48b9b54e053523d23cad"
I20260620 14:50:06.254170 18874 raft_consensus.cc:3037] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:06.254175 18740 raft_consensus.cc:3037] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:06.254808 18874 raft_consensus.cc:2454] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1e3f9d265b46be984f9f9eebda6c38 in term 1.
I20260620 14:50:06.254865 18740 raft_consensus.cc:2454] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1e3f9d265b46be984f9f9eebda6c38 in term 1.
I20260620 14:50:06.254977 18541 leader_election.cc:302] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c1e3f9d265b46be984f9f9eebda6c38, f5858427b8aa48b9b54e053523d23cad; no voters:
I20260620 14:50:06.255108 18940 raft_consensus.cc:2781] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:06.255254 18940 raft_consensus.cc:686] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 [term 1 LEADER]: Becoming Leader. State: Replica: 9c1e3f9d265b46be984f9f9eebda6c38, State: Running, Role: LEADER
I20260620 14:50:06.255344 18940 consensus_queue.cc:237] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5858427b8aa48b9b54e053523d23cad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32873 } } peers { permanent_uuid: "ded4fb0065fe49338987112908fb6513" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44865 } } peers { permanent_uuid: "9c1e3f9d265b46be984f9f9eebda6c38" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } }
I20260620 14:50:06.256179 18460 catalog_manager.cc:5495] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c1e3f9d265b46be984f9f9eebda6c38 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9c1e3f9d265b46be984f9f9eebda6c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5858427b8aa48b9b54e053523d23cad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded4fb0065fe49338987112908fb6513" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1e3f9d265b46be984f9f9eebda6c38" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } health_report { overall_health: HEALTHY } } }
I20260620 14:50:06.262948 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.260738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50222 (local address 127.0.98.1:35185)
0620 14:50:06.260790 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:06.260791 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:06.260799 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:50:06.260843 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:06.260844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:06.260859 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:06.260910 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:06.261003 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.261442 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.262148 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.262350 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.262411 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.262604 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:06.262702 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:06.262776 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:06.262867 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 14:50:06.262876 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:50:06.264612 18874 raft_consensus.cc:1261] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad [term 1 FOLLOWER]: Refusing update from remote peer 9c1e3f9d265b46be984f9f9eebda6c38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:06.264631 18740 raft_consensus.cc:1261] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513 [term 1 FOLLOWER]: Refusing update from remote peer 9c1e3f9d265b46be984f9f9eebda6c38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:06.264925 18940 consensus_queue.cc:1035] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5858427b8aa48b9b54e053523d23cad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:06.265071 18944 consensus_queue.cc:1035] T b3309690fb3843b69ca745dadc3ecc69 P 9c1e3f9d265b46be984f9f9eebda6c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded4fb0065fe49338987112908fb6513" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:06.267406 18948 mvcc.cc:205] Tried to move back new op lower bound from 7298936857657454592 to 7298936857622544384. Current Snapshot: MvccSnapshot[applied={T|T < 7298936857657454592}]
I20260620 14:50:06.268850 18950 mvcc.cc:205] Tried to move back new op lower bound from 7298936857657454592 to 7298936857622544384. Current Snapshot: MvccSnapshot[applied={T|T < 7298936857657454592}]
I20260620 14:50:06.269551 18461 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:06.269644 18461 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:06.270327 18461 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b15f92f5bf004f4997f34b67dc43b674: Sending DeleteTablet for 3 replicas of tablet b3309690fb3843b69ca745dadc3ecc69
I20260620 14:50:06.270659 18586 tablet_service.cc:1514] Processing DeleteTablet for tablet b3309690fb3843b69ca745dadc3ecc69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50218
I20260620 14:50:06.270680 18720 tablet_service.cc:1514] Processing DeleteTablet for tablet b3309690fb3843b69ca745dadc3ecc69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51208
I20260620 14:50:06.270893 18957 tablet_replica.cc:330] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513: stopping tablet replica
I20260620 14:50:06.270895 18853 tablet_service.cc:1514] Processing DeleteTablet for tablet b3309690fb3843b69ca745dadc3ecc69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58678
I20260620 14:50:06.270991 18957 raft_consensus.cc:2229] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:06.271099 18957 raft_consensus.cc:2258] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:06.271202 18958 tablet_replica.cc:330] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad: stopping tablet replica
I20260620 14:50:06.271327 18958 raft_consensus.cc:2229] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:06.271332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 18523
I20260620 14:50:06.271565 18957 ts_tablet_manager.cc:1905] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:06.272246 18958 raft_consensus.cc:2258] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:06.273241 18957 ts_tablet_manager.cc:1918] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:06.273306 18957 log.cc:1196] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/b3309690fb3843b69ca745dadc3ecc69
I20260620 14:50:06.273583 18957 ts_tablet_manager.cc:1939] T b3309690fb3843b69ca745dadc3ecc69 P ded4fb0065fe49338987112908fb6513: Deleting consensus metadata
I20260620 14:50:06.273903 18958 ts_tablet_manager.cc:1905] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:06.273983 18445 catalog_manager.cc:4841] TS ded4fb0065fe49338987112908fb6513 (127.0.98.2:44865): tablet b3309690fb3843b69ca745dadc3ecc69 (table test-table [id=18b0e9fc3c284649ac7aed089c9a620b]) successfully deleted
I20260620 14:50:06.275156 18958 ts_tablet_manager.cc:1918] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:06.275216 18958 log.cc:1196] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/b3309690fb3843b69ca745dadc3ecc69
I20260620 14:50:06.275444 18958 ts_tablet_manager.cc:1939] T b3309690fb3843b69ca745dadc3ecc69 P f5858427b8aa48b9b54e053523d23cad: Deleting consensus metadata
I20260620 14:50:06.275810 18447 catalog_manager.cc:4841] TS f5858427b8aa48b9b54e053523d23cad (127.0.98.3:32873): tablet b3309690fb3843b69ca745dadc3ecc69 (table test-table [id=18b0e9fc3c284649ac7aed089c9a620b]) successfully deleted
W20260620 14:50:06.276665 18448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35185 (error 108)
I20260620 14:50:06.276878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 18657
I20260620 14:50:06.277386 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.276962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35185 (local address 127.0.0.1:50232)
0620 14:50:06.277327 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:06.277361 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346}
W20260620 14:50:06.277556 18448 catalog_manager.cc:4584] TS 9c1e3f9d265b46be984f9f9eebda6c38 (127.0.98.1:35185): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3309690fb3843b69ca745dadc3ecc69: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35185: connect: Connection refused (error 111)
I20260620 14:50:06.281184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 18791
I20260620 14:50:06.284775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 18429
2026-06-20T14:50:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1859 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T14:50:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:06Z Disabled control of system clock
I20260620 14:50:06.309832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40381
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:50:06.387045 18963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:06.387283 18963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:06.387311 18963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:06.388510 18963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:06.388553 18963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:06.388568 18963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:06.388585 18963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:06.388599 18963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:06.389937 18963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40381
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:06.390182 18963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:06.390439 18963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:06.394946 18968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:06.394984 18971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:06.394996 18969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:06.395095 18963 server_base.cc:1034] running on GCE node
I20260620 14:50:06.395308 18963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:06.395601 18963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:06.396739 18963 hybrid_clock.cc:648] HybridClock initialized: now 1781967006396732 us; error 28 us; skew 500 ppm
I20260620 14:50:06.397907 18963 webserver.cc:460] Webserver started at http://127.0.98.62:44911/ using document root <none> and password file <none>
I20260620 14:50:06.398114 18963 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:06.398161 18963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:06.398262 18963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:06.399266 18963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "8ec188787e3b4e37b9e7db3a3c34fa41"
format_stamp: "Formatted at 2026-06-20 14:50:06 on dist-test-slave-cx89"
server_key: "59661ddb2ed324e521b3d6ae43a63a39"
server_key_iv: "d5ddf537fbe190507092b2a299fb9ea3"
server_key_version: "encryptionkey@0"
I20260620 14:50:06.399610 18963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "8ec188787e3b4e37b9e7db3a3c34fa41"
format_stamp: "Formatted at 2026-06-20 14:50:06 on dist-test-slave-cx89"
server_key: "59661ddb2ed324e521b3d6ae43a63a39"
server_key_iv: "d5ddf537fbe190507092b2a299fb9ea3"
server_key_version: "encryptionkey@0"
I20260620 14:50:06.400777 18963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:50:06.401445 18977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:06.401580 18963 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:50:06.401665 18963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "8ec188787e3b4e37b9e7db3a3c34fa41"
format_stamp: "Formatted at 2026-06-20 14:50:06 on dist-test-slave-cx89"
server_key: "59661ddb2ed324e521b3d6ae43a63a39"
server_key_iv: "d5ddf537fbe190507092b2a299fb9ea3"
server_key_version: "encryptionkey@0"
I20260620 14:50:06.401721 18963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:06.411281 18963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:06.411604 18963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:06.411738 18963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:06.415259 18963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40381
I20260620 14:50:06.415293 19029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40381 every 8 connection(s)
I20260620 14:50:06.415609 18963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:06.416472 19030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:06.418591 19030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41: Bootstrap starting.
I20260620 14:50:06.419091 19030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:06.419483 19030 log.cc:824] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:06.420089 19030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41: No bootstrap required, opened a new log
I20260620 14:50:06.421597 19030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec188787e3b4e37b9e7db3a3c34fa41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40381 } }
I20260620 14:50:06.421729 19030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:06.421749 19030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec188787e3b4e37b9e7db3a3c34fa41, State: Initialized, Role: FOLLOWER
I20260620 14:50:06.421854 19030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec188787e3b4e37b9e7db3a3c34fa41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40381 } }
I20260620 14:50:06.421898 19030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:06.421916 19030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:06.421945 19030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:06.422526 19030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec188787e3b4e37b9e7db3a3c34fa41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40381 } }
I20260620 14:50:06.422617 19030 leader_election.cc:302] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ec188787e3b4e37b9e7db3a3c34fa41; no voters:
I20260620 14:50:06.422760 19030 leader_election.cc:288] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:06.422821 19033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:06.422978 19030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:06.422984 19033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41 [term 1 LEADER]: Becoming Leader. State: Replica: 8ec188787e3b4e37b9e7db3a3c34fa41, State: Running, Role: LEADER
I20260620 14:50:06.423106 19033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec188787e3b4e37b9e7db3a3c34fa41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40381 } }
I20260620 14:50:06.423604 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 18963
I20260620 14:50:06.423707 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:06.423921 392 external_mini_cluster.cc:1428] Setting key 734c37f104f90ecf0b99fc84698c1013
I20260620 14:50:06.424136 19034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ec188787e3b4e37b9e7db3a3c34fa41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec188787e3b4e37b9e7db3a3c34fa41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40381 } } }
I20260620 14:50:06.424227 19034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:06.424497 19035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ec188787e3b4e37b9e7db3a3c34fa41. Latest consensus state: current_term: 1 leader_uuid: "8ec188787e3b4e37b9e7db3a3c34fa41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec188787e3b4e37b9e7db3a3c34fa41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40381 } } }
I20260620 14:50:06.424678 19035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41 [sys.catalog]: This master's current role is: LEADER
W20260620 14:50:06.424916 19048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:50:06.424968 19048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:50:06.425026 19051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:06.425163 19051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:06.426642 19051 catalog_manager.cc:1347] Generated new cluster ID: 3af1676aa6f44fe1a7a7f87f7cd7f8ea
I20260620 14:50:06.426728 19051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:06.429999 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.424375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51510 (local address 127.0.98.62:40381)
0620 14:50:06.424587 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:06.424597 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:06.424892 (+ 295us) server_negotiation.cc:407] Connection header received
0620 14:50:06.425018 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:06.425026 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:06.425098 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:06.425225 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:06.425443 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.426717 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.427922 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.428533 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.428652 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.429318 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:06.429322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:06.429322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:06.429516 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:06.429557 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:06.429570 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:06.429628 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:06.429711 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 14:50:06.429787 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":59,"threads_started":1}
I20260620 14:50:06.456087 19051 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:06.456599 19051 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:06.462647 19051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41: Generated new TSK 0
I20260620 14:50:06.462846 19051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:06.467597 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40381
--builtin_ntp_servers=127.0.98.20:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:50:06.558091 19054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:06.558300 19054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:06.558328 19054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:06.559495 19054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:06.559532 19054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:06.559566 19054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:50:06.560881 19054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:06.561115 19054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:06.561381 19054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:06.566057 19060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:06.566095 19062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:06.566169 19054 server_base.cc:1034] running on GCE node
W20260620 14:50:06.566112 19059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:06.566475 19054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:06.566710 19054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:06.567862 19054 hybrid_clock.cc:648] HybridClock initialized: now 1781967006567846 us; error 29 us; skew 500 ppm
I20260620 14:50:06.569058 19054 webserver.cc:460] Webserver started at http://127.0.98.1:35843/ using document root <none> and password file <none>
I20260620 14:50:06.569269 19054 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:06.569319 19054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:06.569442 19054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:06.570470 19054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "aea503b488964165993446fe9b2ed111"
format_stamp: "Formatted at 2026-06-20 14:50:06 on dist-test-slave-cx89"
server_key: "255fb0c3471ec4e2519e4042721e5cb5"
server_key_iv: "96ede27a6187ff8f067e548e4ccdb53d"
server_key_version: "encryptionkey@0"
I20260620 14:50:06.570809 19054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "aea503b488964165993446fe9b2ed111"
format_stamp: "Formatted at 2026-06-20 14:50:06 on dist-test-slave-cx89"
server_key: "255fb0c3471ec4e2519e4042721e5cb5"
server_key_iv: "96ede27a6187ff8f067e548e4ccdb53d"
server_key_version: "encryptionkey@0"
I20260620 14:50:06.572108 19054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:50:06.572781 19068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:06.572922 19054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:06.573004 19054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "aea503b488964165993446fe9b2ed111"
format_stamp: "Formatted at 2026-06-20 14:50:06 on dist-test-slave-cx89"
server_key: "255fb0c3471ec4e2519e4042721e5cb5"
server_key_iv: "96ede27a6187ff8f067e548e4ccdb53d"
server_key_version: "encryptionkey@0"
I20260620 14:50:06.573076 19054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:06.602972 19054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:06.603298 19054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:06.603482 19054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:06.603731 19054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:06.604063 19054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:06.604099 19054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:06.604131 19054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:06.604158 19054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:06.609807 19054 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36525
I20260620 14:50:06.609876 19181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36525 every 8 connection(s)
I20260620 14:50:06.610152 19054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:50:06.613102 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 19054
I20260620 14:50:06.613214 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:50:06.613416 392 external_mini_cluster.cc:1428] Setting key 0f759ae96d34eec87bb46a685834769f
I20260620 14:50:06.614365 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40381
--builtin_ntp_servers=127.0.98.20:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:50:06.615576 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.610796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35679 (local address 127.0.98.62:40381)
0620 14:50:06.610843 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:06.610845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:06.611067 (+ 222us) server_negotiation.cc:407] Connection header received
0620 14:50:06.611586 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:06.611587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:06.611602 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:06.611654 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:06.612315 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.612829 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.614120 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.614321 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.614374 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.615221 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:06.615223 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:06.615223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:06.615228 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:06.615245 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:06.615249 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:06.615295 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:06.615452 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 14:50:06.615491 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:50:06.615794 19184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.610789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40381 (local address 127.0.98.1:35679)
0620 14:50:06.610951 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:06.610984 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:06.611491 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:06.611732 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:06.611736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:06.611747 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:06.612230 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.612235 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.612918 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.612919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.614047 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.614049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.614391 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.614392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.614546 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.615086 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:50:06.615086 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:06.615115 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:50:06.615139 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:06.615326 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:06.615334 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:06.615338 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:50:06.615457 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 14:50:06.615577 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260620 14:50:06.616312 19182 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40381
I20260620 14:50:06.616406 19182 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:06.616600 19182 heartbeater.cc:511] Master 127.0.98.62:40381 requested a full tablet report, sending...
I20260620 14:50:06.617069 18994 ts_manager.cc:194] Registered new tserver with Master: aea503b488964165993446fe9b2ed111 (127.0.98.1:36525)
I20260620 14:50:06.617666 18994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:35679
W20260620 14:50:06.692461 19185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:06.692653 19185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:06.692680 19185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:06.693794 19185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:06.693832 19185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:06.693866 19185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:50:06.695173 19185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:06.695398 19185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:06.695683 19185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:06.700569 19190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:06.700542 19191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:06.700556 19193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:06.700875 19185 server_base.cc:1034] running on GCE node
I20260620 14:50:06.701086 19185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:06.701311 19185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:06.702443 19185 hybrid_clock.cc:648] HybridClock initialized: now 1781967006702423 us; error 35 us; skew 500 ppm
I20260620 14:50:06.703735 19185 webserver.cc:460] Webserver started at http://127.0.98.2:33189/ using document root <none> and password file <none>
I20260620 14:50:06.703945 19185 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:06.704010 19185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:06.704157 19185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:06.705250 19185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "c9bc7f2a29e14f428851caf66e2517fd"
format_stamp: "Formatted at 2026-06-20 14:50:06 on dist-test-slave-cx89"
server_key: "4463cd130fc6d82d677c6b500e1db0df"
server_key_iv: "6235598ef0ce60ac5de3fc8f8397ad71"
server_key_version: "encryptionkey@0"
I20260620 14:50:06.705574 19185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c9bc7f2a29e14f428851caf66e2517fd"
format_stamp: "Formatted at 2026-06-20 14:50:06 on dist-test-slave-cx89"
server_key: "4463cd130fc6d82d677c6b500e1db0df"
server_key_iv: "6235598ef0ce60ac5de3fc8f8397ad71"
server_key_version: "encryptionkey@0"
I20260620 14:50:06.706781 19185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:50:06.707500 19199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:06.707653 19185 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:50:06.707731 19185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "c9bc7f2a29e14f428851caf66e2517fd"
format_stamp: "Formatted at 2026-06-20 14:50:06 on dist-test-slave-cx89"
server_key: "4463cd130fc6d82d677c6b500e1db0df"
server_key_iv: "6235598ef0ce60ac5de3fc8f8397ad71"
server_key_version: "encryptionkey@0"
I20260620 14:50:06.707795 19185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:06.739331 19185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:06.739658 19185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:06.739787 19185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:06.739995 19185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:06.740316 19185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:06.740348 19185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:06.740377 19185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:06.740398 19185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:06.745839 19185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44329
I20260620 14:50:06.745879 19312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44329 every 8 connection(s)
I20260620 14:50:06.746203 19185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:50:06.749361 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 19185
I20260620 14:50:06.749445 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:50:06.749637 392 external_mini_cluster.cc:1428] Setting key 6e49e73925ecf2074d56417a24379af5
I20260620 14:50:06.750516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40381
--builtin_ntp_servers=127.0.98.20:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:50:06.751294 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.746885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39485 (local address 127.0.98.62:40381)
0620 14:50:06.746937 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:06.746939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:06.747164 (+ 225us) server_negotiation.cc:407] Connection header received
0620 14:50:06.747728 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:06.747729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:06.747740 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:06.747788 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:06.748456 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.748972 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.749990 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.750221 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.750277 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.751058 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:06.751060 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:06.751061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:06.751065 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:06.751083 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:06.751087 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:06.751143 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:06.751195 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 14:50:06.751221 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:50:06.751483 19315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.746869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40381 (local address 127.0.98.2:39485)
0620 14:50:06.747050 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:06.747084 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:06.747627 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:06.747869 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:06.747873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:06.747882 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:06.748373 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.748377 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.749060 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.749061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.749915 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.749918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.750296 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.750297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.750415 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.750930 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:50:06.750931 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:06.750959 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:50:06.750982 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:06.751144 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:06.751147 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:06.751150 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:50:06.751195 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 14:50:06.751270 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260620 14:50:06.751995 19313 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40381
I20260620 14:50:06.752090 19313 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:06.752275 19313 heartbeater.cc:511] Master 127.0.98.62:40381 requested a full tablet report, sending...
I20260620 14:50:06.752616 18994 ts_manager.cc:194] Registered new tserver with Master: c9bc7f2a29e14f428851caf66e2517fd (127.0.98.2:44329)
I20260620 14:50:06.753075 18994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:39485
W20260620 14:50:06.833258 19316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:06.833452 19316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:06.833472 19316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:06.834542 19316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:06.834573 19316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:06.834597 19316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:50:06.835913 19316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:06.836092 19316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:06.836335 19316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:06.840905 19321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:06.840905 19322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:06.840942 19316 server_base.cc:1034] running on GCE node
W20260620 14:50:06.841148 19324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:06.841342 19316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:06.841562 19316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:06.842712 19316 hybrid_clock.cc:648] HybridClock initialized: now 1781967006842694 us; error 28 us; skew 500 ppm
I20260620 14:50:06.848201 19316 webserver.cc:460] Webserver started at http://127.0.98.3:42499/ using document root <none> and password file <none>
I20260620 14:50:06.848444 19316 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:06.848493 19316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:06.848654 19316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:06.849704 19316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "892edc381a4542ee93418d191da7e794"
format_stamp: "Formatted at 2026-06-20 14:50:06 on dist-test-slave-cx89"
server_key: "0cc395bdba8906a1949d249492ae916a"
server_key_iv: "5012d035d040d57530640e7f7bc55da6"
server_key_version: "encryptionkey@0"
I20260620 14:50:06.850030 19316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "892edc381a4542ee93418d191da7e794"
format_stamp: "Formatted at 2026-06-20 14:50:06 on dist-test-slave-cx89"
server_key: "0cc395bdba8906a1949d249492ae916a"
server_key_iv: "5012d035d040d57530640e7f7bc55da6"
server_key_version: "encryptionkey@0"
I20260620 14:50:06.851186 19316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:50:06.851907 19330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:06.852052 19316 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:06.852131 19316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "892edc381a4542ee93418d191da7e794"
format_stamp: "Formatted at 2026-06-20 14:50:06 on dist-test-slave-cx89"
server_key: "0cc395bdba8906a1949d249492ae916a"
server_key_iv: "5012d035d040d57530640e7f7bc55da6"
server_key_version: "encryptionkey@0"
I20260620 14:50:06.852198 19316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:06.866489 19316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:06.866842 19316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:06.866978 19316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:06.867223 19316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:06.867573 19316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:06.867609 19316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:06.867630 19316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:06.867646 19316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:06.872913 19316 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36233
I20260620 14:50:06.872959 19443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36233 every 8 connection(s)
I20260620 14:50:06.873252 19316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:50:06.875665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 19316
I20260620 14:50:06.875773 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:50:06.875955 392 external_mini_cluster.cc:1428] Setting key 26e9bf9790a32c8bbeb70ebeb884bb40
I20260620 14:50:06.878260 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.873876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42845 (local address 127.0.98.62:40381)
0620 14:50:06.873935 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:06.873937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:06.874151 (+ 214us) server_negotiation.cc:407] Connection header received
0620 14:50:06.874641 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:06.874642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:06.874654 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:06.874701 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:06.875445 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.876185 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.877126 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.877319 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.877371 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.878080 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:06.878081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:06.878082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:06.878085 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:06.878095 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:06.878098 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:06.878130 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:06.878183 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 14:50:06.878203 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:50:06.878438 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.873858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40381 (local address 127.0.98.3:42845)
0620 14:50:06.874030 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:06.874062 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:06.874546 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:06.874781 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:06.874784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:06.874792 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:06.875340 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.875345 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.876271 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.876272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.877038 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.877040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.877377 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.877377 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.877484 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.877969 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:50:06.877969 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:06.877997 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:50:06.878020 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:06.878132 (+ 112us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:06.878134 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:06.878137 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:50:06.878181 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 14:50:06.878256 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260620 14:50:06.878908 19444 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40381
I20260620 14:50:06.878978 19444 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:06.879122 19444 heartbeater.cc:511] Master 127.0.98.62:40381 requested a full tablet report, sending...
I20260620 14:50:06.879474 18994 ts_manager.cc:194] Registered new tserver with Master: 892edc381a4542ee93418d191da7e794 (127.0.98.3:36233)
I20260620 14:50:06.879791 18994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:42845
I20260620 14:50:06.887513 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:50:06.891531 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.888834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51526 (local address 127.0.98.62:40381)
0620 14:50:06.888873 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:06.888875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:06.888970 (+ 95us) server_negotiation.cc:407] Connection header received
0620 14:50:06.889034 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:06.889035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:06.889046 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:06.889090 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:06.889235 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.889700 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.890401 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.890586 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.890640 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.891249 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:06.891250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:06.891250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:06.891254 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:06.891269 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:06.891273 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:06.891344 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:06.891439 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:50:06.891450 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 14:50:06.892958 18994 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:50:06.894392 18994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:50:06.900179 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.896490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.0.98.1:36525)
0620 14:50:06.896687 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:06.896692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:06.896753 (+ 61us) server_negotiation.cc:407] Connection header received
0620 14:50:06.896854 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:06.896857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:06.896871 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:06.896936 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:06.897077 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.897854 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.899230 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.899806 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.899872 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.899964 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:06.900048 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 14:50:06.900085 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":80,"threads_started":1}
I20260620 14:50:06.900429 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.896254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44329 (local address 127.0.0.1:34266)
0620 14:50:06.896509 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:06.896549 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:06.896610 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:06.896815 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:06.896818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:06.896821 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:06.896900 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.897102 (+ 202us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.898310 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.898311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.899060 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.899063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.899982 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.899984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.900042 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.900046 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:50:06.900074 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 14:50:06.900113 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":85,"threads_started":1}
I20260620 14:50:06.900434 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.896306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41928 (local address 127.0.98.3:36233)
0620 14:50:06.896738 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:06.896742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:06.896752 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:50:06.896787 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:06.896789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:06.896804 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:06.896869 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:06.897029 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.898121 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.899294 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.899982 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.900038 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.900119 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:06.900322 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 14:50:06.900360 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":65,"threads_started":1}
I20260620 14:50:06.900453 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.896476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34266 (local address 127.0.98.2:44329)
0620 14:50:06.896660 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:06.896664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:06.896674 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:50:06.896713 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:06.896715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:06.896731 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:06.896798 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:06.897183 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.898237 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.899140 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.899884 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.900233 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.900322 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:06.900344 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 14:50:06.900379 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":67,"threads_started":1}
I20260620 14:50:06.900504 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.896254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36233 (local address 127.0.0.1:41928)
0620 14:50:06.896516 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:06.896553 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:06.896611 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:06.896925 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:06.896926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:06.896927 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:06.896958 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.896959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.898208 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.898209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.899235 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.899237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.900149 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.900150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.900207 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.900210 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:50:06.900251 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 14:50:06.900285 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":2,"thread_start_us":66,"threads_started":1}
I20260620 14:50:06.900194 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:06.896261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36525 (local address 127.0.0.1:49800)
0620 14:50:06.896705 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:06.896713 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:06.896759 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:06.896953 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:06.896954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:06.896956 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:06.897001 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.897002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.897929 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.897930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.899139 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:06.899142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:06.899880 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:06.899881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:06.899954 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:06.899966 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 14:50:06.900034 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:50:06.900085 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":88,"threads_started":1}
I20260620 14:50:06.901510 19378 tablet_service.cc:1467] Processing CreateTablet for tablet 0058f47c73614957b3aeed291d9858a2 (DEFAULT_TABLE table=test-table [id=5003e1afbc4f441091e22b59d3735853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:06.901480 19247 tablet_service.cc:1467] Processing CreateTablet for tablet 0058f47c73614957b3aeed291d9858a2 (DEFAULT_TABLE table=test-table [id=5003e1afbc4f441091e22b59d3735853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:06.901479 19116 tablet_service.cc:1467] Processing CreateTablet for tablet 0058f47c73614957b3aeed291d9858a2 (DEFAULT_TABLE table=test-table [id=5003e1afbc4f441091e22b59d3735853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:06.901780 19378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0058f47c73614957b3aeed291d9858a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:06.901804 19116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0058f47c73614957b3aeed291d9858a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:06.901803 19247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0058f47c73614957b3aeed291d9858a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:06.904345 19458 tablet_bootstrap.cc:492] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794: Bootstrap starting.
I20260620 14:50:06.904928 19458 tablet_bootstrap.cc:654] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:06.904985 19460 tablet_bootstrap.cc:492] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd: Bootstrap starting.
I20260620 14:50:06.905237 19458 log.cc:824] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:06.905575 19460 tablet_bootstrap.cc:654] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:06.905879 19460 log.cc:824] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:06.905933 19458 tablet_bootstrap.cc:492] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794: No bootstrap required, opened a new log
I20260620 14:50:06.905984 19458 ts_tablet_manager.cc:1397] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:50:06.906221 19459 tablet_bootstrap.cc:492] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111: Bootstrap starting.
I20260620 14:50:06.906514 19460 tablet_bootstrap.cc:492] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd: No bootstrap required, opened a new log
I20260620 14:50:06.906574 19460 ts_tablet_manager.cc:1397] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:50:06.906836 19459 tablet_bootstrap.cc:654] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:06.907131 19459 log.cc:824] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:06.907866 19459 tablet_bootstrap.cc:492] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111: No bootstrap required, opened a new log
I20260620 14:50:06.907933 19459 ts_tablet_manager.cc:1397] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:50:06.908424 19458 raft_consensus.cc:348] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892edc381a4542ee93418d191da7e794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36233 } } peers { permanent_uuid: "aea503b488964165993446fe9b2ed111" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36525 } } peers { permanent_uuid: "c9bc7f2a29e14f428851caf66e2517fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44329 } }
I20260620 14:50:06.908614 19458 raft_consensus.cc:374] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:06.908550 19460 raft_consensus.cc:348] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892edc381a4542ee93418d191da7e794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36233 } } peers { permanent_uuid: "aea503b488964165993446fe9b2ed111" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36525 } } peers { permanent_uuid: "c9bc7f2a29e14f428851caf66e2517fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44329 } }
I20260620 14:50:06.908656 19458 raft_consensus.cc:729] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 892edc381a4542ee93418d191da7e794, State: Initialized, Role: FOLLOWER
I20260620 14:50:06.908689 19460 raft_consensus.cc:374] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:06.908727 19460 raft_consensus.cc:729] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9bc7f2a29e14f428851caf66e2517fd, State: Initialized, Role: FOLLOWER
I20260620 14:50:06.908785 19458 consensus_queue.cc:260] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892edc381a4542ee93418d191da7e794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36233 } } peers { permanent_uuid: "aea503b488964165993446fe9b2ed111" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36525 } } peers { permanent_uuid: "c9bc7f2a29e14f428851caf66e2517fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44329 } }
I20260620 14:50:06.908829 19460 consensus_queue.cc:260] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892edc381a4542ee93418d191da7e794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36233 } } peers { permanent_uuid: "aea503b488964165993446fe9b2ed111" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36525 } } peers { permanent_uuid: "c9bc7f2a29e14f428851caf66e2517fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44329 } }
I20260620 14:50:06.909034 19458 ts_tablet_manager.cc:1428] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:06.909073 19460 ts_tablet_manager.cc:1428] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:50:06.909161 19444 heartbeater.cc:503] Master 127.0.98.62:40381 was elected leader, sending a full tablet report...
I20260620 14:50:06.909412 19313 heartbeater.cc:503] Master 127.0.98.62:40381 was elected leader, sending a full tablet report...
I20260620 14:50:06.910130 19459 raft_consensus.cc:348] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892edc381a4542ee93418d191da7e794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36233 } } peers { permanent_uuid: "aea503b488964165993446fe9b2ed111" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36525 } } peers { permanent_uuid: "c9bc7f2a29e14f428851caf66e2517fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44329 } }
I20260620 14:50:06.910274 19459 raft_consensus.cc:374] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:06.910315 19459 raft_consensus.cc:729] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea503b488964165993446fe9b2ed111, State: Initialized, Role: FOLLOWER
I20260620 14:50:06.910426 19459 consensus_queue.cc:260] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892edc381a4542ee93418d191da7e794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36233 } } peers { permanent_uuid: "aea503b488964165993446fe9b2ed111" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36525 } } peers { permanent_uuid: "c9bc7f2a29e14f428851caf66e2517fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44329 } }
I20260620 14:50:06.910619 19459 ts_tablet_manager.cc:1428] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:06.910682 19182 heartbeater.cc:503] Master 127.0.98.62:40381 was elected leader, sending a full tablet report...
W20260620 14:50:06.996728 19314 tablet.cc:2369] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:50:07.060935 19466 raft_consensus.cc:482] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:50:07.061062 19466 raft_consensus.cc:504] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892edc381a4542ee93418d191da7e794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36233 } } peers { permanent_uuid: "aea503b488964165993446fe9b2ed111" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36525 } } peers { permanent_uuid: "c9bc7f2a29e14f428851caf66e2517fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44329 } }
I20260620 14:50:07.061381 19466 leader_election.cc:288] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 892edc381a4542ee93418d191da7e794 (127.0.98.3:36233), c9bc7f2a29e14f428851caf66e2517fd (127.0.98.2:44329)
I20260620 14:50:07.064055 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:07.061654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51839 (local address 127.0.98.3:36233)
0620 14:50:07.061731 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:07.061733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:07.061742 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:50:07.061771 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:07.061771 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:07.061778 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:07.061823 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:07.061960 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:07.062433 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:07.063334 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:07.063744 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:07.063807 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:07.063817 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:07.063947 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 14:50:07.063982 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:50:07.064083 19184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:07.061551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36233 (local address 127.0.98.1:51839)
0620 14:50:07.061623 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:07.061635 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:07.061686 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:07.061835 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:07.061836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:07.061838 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:07.061903 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:07.061905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:07.062503 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:07.062504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:07.063250 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:07.063252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:07.063818 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:07.063819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:07.063888 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:07.063891 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:50:07.063935 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 14:50:07.063979 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 14:50:07.064103 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:07.061626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44267 (local address 127.0.98.2:44329)
0620 14:50:07.061680 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:07.061682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:07.061816 (+ 134us) server_negotiation.cc:407] Connection header received
0620 14:50:07.061869 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:07.061870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:07.061882 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:07.061929 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:07.062067 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:07.062530 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:07.063439 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:07.063793 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:07.063853 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:07.063862 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:07.063995 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 14:50:07.064029 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:50:07.064411 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:07.061551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44329 (local address 127.0.98.1:44267)
0620 14:50:07.061735 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:07.061744 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:07.061814 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:07.061936 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:07.061937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:07.061938 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:07.062005 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:07.062007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:07.062605 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:07.062606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:07.063325 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:07.063327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:07.063862 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:07.063863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:07.063933 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:07.063936 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:50:07.063980 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 14:50:07.064017 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":15,"thread_start_us":97,"threads_started":1}
I20260620 14:50:07.064405 19398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0058f47c73614957b3aeed291d9858a2" candidate_uuid: "aea503b488964165993446fe9b2ed111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "892edc381a4542ee93418d191da7e794" is_pre_election: true
I20260620 14:50:07.064563 19398 raft_consensus.cc:2454] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aea503b488964165993446fe9b2ed111 in term 0.
I20260620 14:50:07.064746 19267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0058f47c73614957b3aeed291d9858a2" candidate_uuid: "aea503b488964165993446fe9b2ed111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9bc7f2a29e14f428851caf66e2517fd" is_pre_election: true
I20260620 14:50:07.064808 19072 leader_election.cc:302] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 892edc381a4542ee93418d191da7e794, aea503b488964165993446fe9b2ed111; no voters:
I20260620 14:50:07.064879 19267 raft_consensus.cc:2454] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aea503b488964165993446fe9b2ed111 in term 0.
I20260620 14:50:07.064914 19466 raft_consensus.cc:2781] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:50:07.064962 19466 raft_consensus.cc:482] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:50:07.064978 19466 raft_consensus.cc:3037] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:07.065623 19466 raft_consensus.cc:504] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892edc381a4542ee93418d191da7e794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36233 } } peers { permanent_uuid: "aea503b488964165993446fe9b2ed111" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36525 } } peers { permanent_uuid: "c9bc7f2a29e14f428851caf66e2517fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44329 } }
I20260620 14:50:07.065732 19466 leader_election.cc:288] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [CANDIDATE]: Term 1 election: Requested vote from peers 892edc381a4542ee93418d191da7e794 (127.0.98.3:36233), c9bc7f2a29e14f428851caf66e2517fd (127.0.98.2:44329)
I20260620 14:50:07.065883 19398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0058f47c73614957b3aeed291d9858a2" candidate_uuid: "aea503b488964165993446fe9b2ed111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "892edc381a4542ee93418d191da7e794"
I20260620 14:50:07.065896 19267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0058f47c73614957b3aeed291d9858a2" candidate_uuid: "aea503b488964165993446fe9b2ed111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9bc7f2a29e14f428851caf66e2517fd"
I20260620 14:50:07.065950 19398 raft_consensus.cc:3037] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:07.065964 19267 raft_consensus.cc:3037] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:07.066620 19398 raft_consensus.cc:2454] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea503b488964165993446fe9b2ed111 in term 1.
I20260620 14:50:07.066699 19267 raft_consensus.cc:2454] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea503b488964165993446fe9b2ed111 in term 1.
I20260620 14:50:07.066776 19072 leader_election.cc:302] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 892edc381a4542ee93418d191da7e794, aea503b488964165993446fe9b2ed111; no voters:
I20260620 14:50:07.066855 19466 raft_consensus.cc:2781] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:07.066989 19466 raft_consensus.cc:686] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [term 1 LEADER]: Becoming Leader. State: Replica: aea503b488964165993446fe9b2ed111, State: Running, Role: LEADER
I20260620 14:50:07.067080 19466 consensus_queue.cc:237] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892edc381a4542ee93418d191da7e794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36233 } } peers { permanent_uuid: "aea503b488964165993446fe9b2ed111" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36525 } } peers { permanent_uuid: "c9bc7f2a29e14f428851caf66e2517fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44329 } }
I20260620 14:50:07.067909 18994 catalog_manager.cc:5495] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 reported cstate change: term changed from 0 to 1, leader changed from <none> to aea503b488964165993446fe9b2ed111 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "aea503b488964165993446fe9b2ed111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892edc381a4542ee93418d191da7e794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aea503b488964165993446fe9b2ed111" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9bc7f2a29e14f428851caf66e2517fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44329 } health_report { overall_health: UNKNOWN } } }
I20260620 14:50:07.084409 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:07.081902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49802 (local address 127.0.98.1:36525)
0620 14:50:07.081970 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:07.081972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:07.081981 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:50:07.082023 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:07.082024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:07.082045 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:07.082113 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:07.082275 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:07.082790 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:07.083703 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:07.083896 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:07.083950 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:07.084060 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:07.084148 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:07.084229 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:07.084330 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 14:50:07.084339 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:50:07.086112 19267 raft_consensus.cc:1261] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd [term 1 FOLLOWER]: Refusing update from remote peer aea503b488964165993446fe9b2ed111: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:07.086110 19398 raft_consensus.cc:1261] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794 [term 1 FOLLOWER]: Refusing update from remote peer aea503b488964165993446fe9b2ed111: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:07.086481 19468 consensus_queue.cc:1035] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [LEADER]: Connected to new peer: Peer: permanent_uuid: "892edc381a4542ee93418d191da7e794" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:07.086619 19466 consensus_queue.cc:1035] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9bc7f2a29e14f428851caf66e2517fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:07.089066 19471 mvcc.cc:205] Tried to move back new op lower bound from 7298936861022142464 to 7298936860947247104. Current Snapshot: MvccSnapshot[applied={T|T < 7298936861022142464}]
I20260620 14:50:07.091356 18994 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:07.091470 18994 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:07.092190 18994 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ec188787e3b4e37b9e7db3a3c34fa41: Sending DeleteTablet for 3 replicas of tablet 0058f47c73614957b3aeed291d9858a2
I20260620 14:50:07.092525 19116 tablet_service.cc:1514] Processing DeleteTablet for tablet 0058f47c73614957b3aeed291d9858a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:07 UTC) from {username='slave'} at 127.0.0.1:49800
I20260620 14:50:07.092542 19247 tablet_service.cc:1514] Processing DeleteTablet for tablet 0058f47c73614957b3aeed291d9858a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:07 UTC) from {username='slave'} at 127.0.0.1:34266
I20260620 14:50:07.092713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 19054
I20260620 14:50:07.092801 19481 tablet_replica.cc:330] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111: stopping tablet replica
I20260620 14:50:07.092840 19378 tablet_service.cc:1514] Processing DeleteTablet for tablet 0058f47c73614957b3aeed291d9858a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:07 UTC) from {username='slave'} at 127.0.0.1:41928
I20260620 14:50:07.092888 19481 raft_consensus.cc:2229] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:50:07.092960 19481 raft_consensus.cc:2258] T 0058f47c73614957b3aeed291d9858a2 P aea503b488964165993446fe9b2ed111 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:07.093251 19482 tablet_replica.cc:330] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd: stopping tablet replica
I20260620 14:50:07.093329 19482 raft_consensus.cc:2229] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:07.093411 19482 raft_consensus.cc:2258] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:07.094008 19482 ts_tablet_manager.cc:1905] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:07.095311 19483 tablet_replica.cc:330] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794: stopping tablet replica
I20260620 14:50:07.095432 19483 raft_consensus.cc:2229] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:07.095476 19482 ts_tablet_manager.cc:1918] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:07.095544 19482 log.cc:1196] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/0058f47c73614957b3aeed291d9858a2
I20260620 14:50:07.095539 19483 raft_consensus.cc:2258] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:07.095789 19482 ts_tablet_manager.cc:1939] T 0058f47c73614957b3aeed291d9858a2 P c9bc7f2a29e14f428851caf66e2517fd: Deleting consensus metadata
I20260620 14:50:07.096153 18979 catalog_manager.cc:4841] TS c9bc7f2a29e14f428851caf66e2517fd (127.0.98.2:44329): tablet 0058f47c73614957b3aeed291d9858a2 (table test-table [id=5003e1afbc4f441091e22b59d3735853]) successfully deleted
I20260620 14:50:07.096514 19483 ts_tablet_manager.cc:1905] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:07.097697 19483 ts_tablet_manager.cc:1918] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:07.097755 19483 log.cc:1196] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/0058f47c73614957b3aeed291d9858a2
I20260620 14:50:07.097934 19483 ts_tablet_manager.cc:1939] T 0058f47c73614957b3aeed291d9858a2 P 892edc381a4542ee93418d191da7e794: Deleting consensus metadata
I20260620 14:50:07.098323 18981 catalog_manager.cc:4841] TS 892edc381a4542ee93418d191da7e794 (127.0.98.3:36233): tablet 0058f47c73614957b3aeed291d9858a2 (table test-table [id=5003e1afbc4f441091e22b59d3735853]) successfully deleted
W20260620 14:50:07.098475 18978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36525 (error 108)
I20260620 14:50:07.098709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 19185
I20260620 14:50:07.098863 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:07.098764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36525 (local address 127.0.0.1:49806)
0620 14:50:07.098806 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:07.098837 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 14:50:07.098951 18978 catalog_manager.cc:4584] TS aea503b488964165993446fe9b2ed111 (127.0.98.1:36525): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0058f47c73614957b3aeed291d9858a2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36525: connect: Connection refused (error 111)
I20260620 14:50:07.102522 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 19316
I20260620 14:50:07.106284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 18963
2026-06-20T14:50:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (820 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:07 dist-test-slave-cx89 krb5kdc[19487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:07 dist-test-slave-cx89 krb5kdc[19487](info): set up 2 sockets
Jun 20 14:50:07 dist-test-slave-cx89 krb5kdc[19487](info): commencing operation
krb5kdc: starting...
W20260620 14:50:08.063050 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:08 dist-test-slave-cx89 krb5kdc[19487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:50:08.171392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43567
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:08.250663 19500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:08.250866 19500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:08.250896 19500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:08.252058 19500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:08.252101 19500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:08.252127 19500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:08.252146 19500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:08.252166 19500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:08.253625 19500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:08.253855 19500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:08.254079 19500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:08.258826 19505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:08.258836 19508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:08.258857 19506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:08.258884 19500 server_base.cc:1034] running on GCE node
I20260620 14:50:08.259228 19500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:08.259531 19500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:08.260679 19500 hybrid_clock.cc:648] HybridClock initialized: now 1781967008260662 us; error 28 us; skew 500 ppm
Jun 20 14:50:08 dist-test-slave-cx89 krb5kdc[19487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:08.263084 19500 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:50:08.263567 19500 webserver.cc:460] Webserver started at http://127.0.98.62:43227/ using document root <none> and password file <none>
I20260620 14:50:08.263801 19500 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:08.263854 19500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:08.263988 19500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:08.265146 19500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "955655352a6646cab0e3833d1c878cbf"
format_stamp: "Formatted at 2026-06-20 14:50:08 on dist-test-slave-cx89"
server_key: "ba16cc8ed3215df4e9ede7e333d45142"
server_key_iv: "0c2dd262b005feae7fe8f647b719afcc"
server_key_version: "encryptionkey@0"
I20260620 14:50:08.265506 19500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "955655352a6646cab0e3833d1c878cbf"
format_stamp: "Formatted at 2026-06-20 14:50:08 on dist-test-slave-cx89"
server_key: "ba16cc8ed3215df4e9ede7e333d45142"
server_key_iv: "0c2dd262b005feae7fe8f647b719afcc"
server_key_version: "encryptionkey@0"
I20260620 14:50:08.266819 19500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:50:08.267496 19515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:08.267647 19500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:08.267726 19500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "955655352a6646cab0e3833d1c878cbf"
format_stamp: "Formatted at 2026-06-20 14:50:08 on dist-test-slave-cx89"
server_key: "ba16cc8ed3215df4e9ede7e333d45142"
server_key_iv: "0c2dd262b005feae7fe8f647b719afcc"
server_key_version: "encryptionkey@0"
I20260620 14:50:08.267793 19500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:08.285959 19500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:08.287971 19500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:08.288105 19500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:08.291747 19500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43567
I20260620 14:50:08.291787 19567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43567 every 8 connection(s)
I20260620 14:50:08.292107 19500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:08.292923 19568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:08.295143 19568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf: Bootstrap starting.
I20260620 14:50:08.295589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 19500
I20260620 14:50:08.295684 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:08.295852 392 external_mini_cluster.cc:1428] Setting key 903ce6a4f90b77dec3c7cdc919fe7b68
I20260620 14:50:08.295866 19568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:08.296186 19568 log.cc:824] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:08.296929 19568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf: No bootstrap required, opened a new log
I20260620 14:50:08.298991 19568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955655352a6646cab0e3833d1c878cbf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43567 } }
I20260620 14:50:08.299172 19568 raft_consensus.cc:374] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:08.299206 19568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 955655352a6646cab0e3833d1c878cbf, State: Initialized, Role: FOLLOWER
I20260620 14:50:08.299331 19568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955655352a6646cab0e3833d1c878cbf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43567 } }
I20260620 14:50:08.299403 19568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:08.299474 19568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:08.299517 19568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:08.300104 19568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955655352a6646cab0e3833d1c878cbf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43567 } }
I20260620 14:50:08.300199 19568 leader_election.cc:302] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 955655352a6646cab0e3833d1c878cbf; no voters:
I20260620 14:50:08.300344 19568 leader_election.cc:288] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:08.300393 19573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:08.300580 19573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf [term 1 LEADER]: Becoming Leader. State: Replica: 955655352a6646cab0e3833d1c878cbf, State: Running, Role: LEADER
I20260620 14:50:08.300653 19568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:08.300719 19573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955655352a6646cab0e3833d1c878cbf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43567 } }
I20260620 14:50:08.301069 19574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "955655352a6646cab0e3833d1c878cbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955655352a6646cab0e3833d1c878cbf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43567 } } }
I20260620 14:50:08.301080 19575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 955655352a6646cab0e3833d1c878cbf. Latest consensus state: current_term: 1 leader_uuid: "955655352a6646cab0e3833d1c878cbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955655352a6646cab0e3833d1c878cbf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43567 } } }
I20260620 14:50:08.301163 19574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:08.301187 19575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:08.301887 19581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:08.302487 19581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 14:50:08 dist-test-slave-cx89 krb5kdc[19487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:50:08.304288 19581 catalog_manager.cc:1347] Generated new cluster ID: 8163eda3e6f04368a8e238c35ce97084
I20260620 14:50:08.304342 19581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:08.306624 19572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.296340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.0.98.62:43567)
0620 14:50:08.296603 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:08.296612 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:08.296647 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:50:08.297115 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:08.297123 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:08.297182 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:08.297300 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:08.298007 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.298844 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.299860 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.300316 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.300418 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:08.303628 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:08.303633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:08.303633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:08.303645 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:08.305245 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:08.305559 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:08.305561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:08.305562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:08.305651 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:08.305816 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:08.305816 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:08.305817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:08.306046 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:08.306215 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:08.306288 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 14:50:08.306418 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
I20260620 14:50:08.310127 19581 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:08.310825 19581 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:08.317039 19581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf: Generated new TSK 0
I20260620 14:50:08.317210 19581 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:50:08.345726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43567
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:08.429621 19594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:08.429831 19594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:08.429853 19594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:08.431015 19594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:08.431057 19594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:08.431082 19594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:50:08.432587 19594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:08.432816 19594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:08.433024 19594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:08.437762 19602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:08.437774 19599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:08.437749 19600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:08.437947 19594 server_base.cc:1034] running on GCE node
I20260620 14:50:08.438202 19594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:08.438443 19594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:08.439585 19594 hybrid_clock.cc:648] HybridClock initialized: now 1781967008439568 us; error 29 us; skew 500 ppm
Jun 20 14:50:08 dist-test-slave-cx89 krb5kdc[19487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:08.441941 19594 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:50:08.442365 19594 webserver.cc:460] Webserver started at http://127.0.98.1:43871/ using document root <none> and password file <none>
I20260620 14:50:08.442569 19594 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:08.442611 19594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:08.442723 19594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:08.443789 19594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "345c3386a2ff40008119a74cba28be0b"
format_stamp: "Formatted at 2026-06-20 14:50:08 on dist-test-slave-cx89"
server_key: "0ce04e181520f9b17b495dca30b70500"
server_key_iv: "3bc5acb36f8292dd2373f6219aefd17a"
server_key_version: "encryptionkey@0"
I20260620 14:50:08.444108 19594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "345c3386a2ff40008119a74cba28be0b"
format_stamp: "Formatted at 2026-06-20 14:50:08 on dist-test-slave-cx89"
server_key: "0ce04e181520f9b17b495dca30b70500"
server_key_iv: "3bc5acb36f8292dd2373f6219aefd17a"
server_key_version: "encryptionkey@0"
I20260620 14:50:08.445358 19594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:50:08.446077 19609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:08.446236 19594 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:08.446316 19594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "345c3386a2ff40008119a74cba28be0b"
format_stamp: "Formatted at 2026-06-20 14:50:08 on dist-test-slave-cx89"
server_key: "0ce04e181520f9b17b495dca30b70500"
server_key_iv: "3bc5acb36f8292dd2373f6219aefd17a"
server_key_version: "encryptionkey@0"
I20260620 14:50:08.446374 19594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:08.455013 19594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:08.456983 19594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:08.457105 19594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:08.457293 19594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:08.457564 19594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:08.457595 19594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:08.457623 19594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:08.457645 19594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:08.462826 19594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39065
I20260620 14:50:08.462883 19722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39065 every 8 connection(s)
I20260620 14:50:08.463143 19594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:50:08 dist-test-slave-cx89 krb5kdc[19487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:50:08.470582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 19594
I20260620 14:50:08.470688 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:50:08.470911 392 external_mini_cluster.cc:1428] Setting key 26ca64323f0ad39b516377e01a9d2f2a
I20260620 14:50:08.471774 19572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.463821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39623 (local address 127.0.98.62:43567)
0620 14:50:08.463879 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:08.463881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:08.464059 (+ 178us) server_negotiation.cc:407] Connection header received
0620 14:50:08.464927 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:08.464928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:08.464936 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:08.464990 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:08.465523 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.466010 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.466937 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.467142 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.467195 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:08.469101 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:08.469103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:08.469104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:08.469111 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:08.470320 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:08.470668 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:08.470670 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:08.470670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:08.470727 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:08.471043 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:08.471044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:08.471044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:08.471177 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:08.471247 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:08.471616 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 14:50:08.471709 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:50:08.471895 19725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.463788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43567 (local address 127.0.98.1:39623)
0620 14:50:08.463940 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:08.463979 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:08.464848 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:08.465042 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:08.465045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:08.465281 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:08.465461 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:08.465466 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.466068 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.466069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:08.466848 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:08.466850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.467202 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.467203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:08.467294 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:08.467746 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:08.467753 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:08.469014 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:08.470390 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:08.470397 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:08.470400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:08.470605 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:08.470787 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:08.470788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:08.470788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:08.470937 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:08.471234 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:08.471241 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:08.471397 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 14:50:08.471527 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 14:50:08.471669 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
I20260620 14:50:08.472321 19723 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43567
I20260620 14:50:08.472409 19723 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:08.472576 19723 heartbeater.cc:511] Master 127.0.98.62:43567 requested a full tablet report, sending...
I20260620 14:50:08.473065 19532 ts_manager.cc:194] Registered new tserver with Master: 345c3386a2ff40008119a74cba28be0b (127.0.98.1:39065)
I20260620 14:50:08.473659 19532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39623
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:50:08.493746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43567
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:08.575212 19728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:08.575448 19728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:08.575484 19728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:08.576623 19728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:08.576668 19728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:08.576705 19728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:50:08.578258 19728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:08.578534 19728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:08.578763 19728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:08.583643 19733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:08.583655 19728 server_base.cc:1034] running on GCE node
W20260620 14:50:08.583709 19736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:08.583752 19734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:08.584093 19728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:08.584332 19728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:08.585471 19728 hybrid_clock.cc:648] HybridClock initialized: now 1781967008585459 us; error 30 us; skew 500 ppm
Jun 20 14:50:08 dist-test-slave-cx89 krb5kdc[19487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:08.587944 19728 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:50:08.588362 19728 webserver.cc:460] Webserver started at http://127.0.98.2:37081/ using document root <none> and password file <none>
I20260620 14:50:08.588565 19728 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:08.588613 19728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:08.588732 19728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:08.589744 19728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "fef701d7c3ea4700801deb826a832e18"
format_stamp: "Formatted at 2026-06-20 14:50:08 on dist-test-slave-cx89"
server_key: "feb408f18b9e400763bd2698cdd30207"
server_key_iv: "6b5d9c5b3a3a0b97fb161e9abe190a57"
server_key_version: "encryptionkey@0"
I20260620 14:50:08.590072 19728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fef701d7c3ea4700801deb826a832e18"
format_stamp: "Formatted at 2026-06-20 14:50:08 on dist-test-slave-cx89"
server_key: "feb408f18b9e400763bd2698cdd30207"
server_key_iv: "6b5d9c5b3a3a0b97fb161e9abe190a57"
server_key_version: "encryptionkey@0"
I20260620 14:50:08.591236 19728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:50:08.591917 19743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:08.592060 19728 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:08.592139 19728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "fef701d7c3ea4700801deb826a832e18"
format_stamp: "Formatted at 2026-06-20 14:50:08 on dist-test-slave-cx89"
server_key: "feb408f18b9e400763bd2698cdd30207"
server_key_iv: "6b5d9c5b3a3a0b97fb161e9abe190a57"
server_key_version: "encryptionkey@0"
I20260620 14:50:08.592203 19728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:08.609277 19728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:08.611378 19728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:08.611538 19728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:08.611742 19728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:08.612033 19728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:08.612064 19728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:08.612092 19728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:08.612109 19728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:08.617478 19728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42801
I20260620 14:50:08.617534 19856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42801 every 8 connection(s)
I20260620 14:50:08.617823 19728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:50:08.618221 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 19728
I20260620 14:50:08.618328 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:50:08.618520 392 external_mini_cluster.cc:1428] Setting key d49e22dba1b46a2d49970cb2e7f9282d
Jun 20 14:50:08 dist-test-slave-cx89 krb5kdc[19487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:50:08.627506 19572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.618484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41681 (local address 127.0.98.62:43567)
0620 14:50:08.618539 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:08.618542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:08.618730 (+ 188us) server_negotiation.cc:407] Connection header received
0620 14:50:08.619783 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:08.619784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:08.619796 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:08.619863 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:08.620526 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.621016 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.622018 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.622212 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.622277 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:08.624771 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:08.624773 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:08.624774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:08.624781 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:08.626180 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:08.626568 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:08.626569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:08.626570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:08.626623 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:08.626841 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:08.626842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:08.626843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:08.626948 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:08.627002 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:08.627302 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 14:50:08.627385 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:50:08.627681 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.618468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43567 (local address 127.0.98.2:41681)
0620 14:50:08.618606 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:08.618647 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:08.619680 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:08.619920 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:08.619923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:08.620168 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:08.620442 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:08.620448 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.621097 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.621098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:08.621939 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:08.621941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.622313 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.622315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:08.622444 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:08.622966 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:08.622973 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:08.624678 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:08.626281 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:08.626289 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:08.626291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:08.626504 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:08.626677 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:08.626678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:08.626678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:08.626788 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:08.627002 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:08.627008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:08.627144 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 14:50:08.627274 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 14:50:08.627463 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20260620 14:50:08.628139 19857 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43567
I20260620 14:50:08.628237 19857 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:08.628407 19857 heartbeater.cc:511] Master 127.0.98.62:43567 requested a full tablet report, sending...
I20260620 14:50:08.628762 19532 ts_manager.cc:194] Registered new tserver with Master: fef701d7c3ea4700801deb826a832e18 (127.0.98.2:42801)
I20260620 14:50:08.629209 19532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:50:08.644166 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43567
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:08.722107 19862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:08.722317 19862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:08.722361 19862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:08.723511 19862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:08.723556 19862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:08.723591 19862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:50:08.725026 19862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:08.725251 19862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:08.725449 19862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:08.730017 19868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:08.730018 19870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:08.730018 19867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:08.730154 19862 server_base.cc:1034] running on GCE node
I20260620 14:50:08.730427 19862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:08.730669 19862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:08.731820 19862 hybrid_clock.cc:648] HybridClock initialized: now 1781967008731804 us; error 25 us; skew 500 ppm
Jun 20 14:50:08 dist-test-slave-cx89 krb5kdc[19487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:08.734306 19862 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:50:08.734714 19862 webserver.cc:460] Webserver started at http://127.0.98.3:44407/ using document root <none> and password file <none>
I20260620 14:50:08.734906 19862 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:08.734951 19862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:08.735069 19862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:08.736101 19862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "4df22153b6e0476a8163586faca56b04"
format_stamp: "Formatted at 2026-06-20 14:50:08 on dist-test-slave-cx89"
server_key: "8b762bf8d80c0aaf865f6f9c079bffbf"
server_key_iv: "97121387788ed738d3dc801c41a46999"
server_key_version: "encryptionkey@0"
I20260620 14:50:08.736408 19862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4df22153b6e0476a8163586faca56b04"
format_stamp: "Formatted at 2026-06-20 14:50:08 on dist-test-slave-cx89"
server_key: "8b762bf8d80c0aaf865f6f9c079bffbf"
server_key_iv: "97121387788ed738d3dc801c41a46999"
server_key_version: "encryptionkey@0"
I20260620 14:50:08.737582 19862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:50:08.738238 19877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:08.738428 19862 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:08.738513 19862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "4df22153b6e0476a8163586faca56b04"
format_stamp: "Formatted at 2026-06-20 14:50:08 on dist-test-slave-cx89"
server_key: "8b762bf8d80c0aaf865f6f9c079bffbf"
server_key_iv: "97121387788ed738d3dc801c41a46999"
server_key_version: "encryptionkey@0"
I20260620 14:50:08.738570 19862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:08.756837 19862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:08.758831 19862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:08.758970 19862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:08.759181 19862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:08.759506 19862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:08.759542 19862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:08.759572 19862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:08.759591 19862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:08.766254 19862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41117
I20260620 14:50:08.766353 19990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41117 every 8 connection(s)
I20260620 14:50:08.766656 19862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:50:08.768716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 19862
I20260620 14:50:08.768821 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:50:08.768985 392 external_mini_cluster.cc:1428] Setting key a15c01d2f2262085ac7545b62db1d595
Jun 20 14:50:08 dist-test-slave-cx89 krb5kdc[19487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:50:08.775635 19572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.767365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58191 (local address 127.0.98.62:43567)
0620 14:50:08.767406 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:08.767408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:08.767628 (+ 220us) server_negotiation.cc:407] Connection header received
0620 14:50:08.768539 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:08.768541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:08.768555 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:08.768648 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:08.769347 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.770011 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.771062 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.771257 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.771305 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:08.773249 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:08.773250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:08.773250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:08.773257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:08.774436 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:08.774773 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:08.774774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:08.774774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:08.774819 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:08.774993 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:08.774993 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:08.774993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:08.775083 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:08.775133 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:08.775500 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 14:50:08.775567 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:50:08.775800 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.767328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43567 (local address 127.0.98.3:58191)
0620 14:50:08.767515 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:08.767549 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:08.768423 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:08.768701 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:08.768704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:08.768991 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:08.769268 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:08.769275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.770107 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.770108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:08.770985 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:08.770988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.771330 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.771331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:08.771472 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:08.771905 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:08.771914 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:08.773161 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:08.774502 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:08.774507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:08.774509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:08.774717 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:08.774862 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:08.774863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:08.774863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:08.774952 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:08.775127 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:08.775129 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:08.775304 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 14:50:08.775443 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 14:50:08.775582 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":115,"threads_started":1}
I20260620 14:50:08.776240 19991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43567
I20260620 14:50:08.776324 19991 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:08.776485 19991 heartbeater.cc:511] Master 127.0.98.62:43567 requested a full tablet report, sending...
I20260620 14:50:08.776813 19532 ts_manager.cc:194] Registered new tserver with Master: 4df22153b6e0476a8163586faca56b04 (127.0.98.3:41117)
I20260620 14:50:08.777165 19532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58191
I20260620 14:50:08.780555 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:50:08.787223 19572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.781858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55992 (local address 127.0.98.62:43567)
0620 14:50:08.781923 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:08.781925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:08.781998 (+ 73us) server_negotiation.cc:407] Connection header received
0620 14:50:08.782053 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:08.782054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:08.782062 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:08.782105 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:08.782730 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.783194 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.783999 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.784185 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.784233 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:08.785348 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:08.785348 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:08.785349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:08.785354 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:08.786406 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:08.786669 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:08.786670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:08.786670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:08.786714 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:08.786835 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:08.786835 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:08.786836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:08.786956 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:08.787022 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:08.787093 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 14:50:08.787152 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:50:08.788697 19532 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:50:08.790194 19532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:50:08.796242 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.792244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41117 (local address 127.0.0.1:48536)
0620 14:50:08.792638 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:08.792649 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:08.792693 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:08.793000 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:08.793001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:08.793003 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:08.793044 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:08.793045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.794287 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.794289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:08.795143 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:08.795145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.796034 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.796035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:08.796093 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:08.796103 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 14:50:08.796152 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:50:08.796188 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":108,"threads_started":1}
I20260620 14:50:08.796280 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.792310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.0.98.3:41117)
0620 14:50:08.792857 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:08.792861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:08.792869 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:50:08.792906 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:08.792907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:08.792922 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:08.792998 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:08.793107 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.794205 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.795223 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.795961 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.796027 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:08.796083 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:08.796166 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 14:50:08.796201 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":66,"threads_started":1}
I20260620 14:50:08.796890 19925 tablet_service.cc:1467] Processing CreateTablet for tablet c23463b0681945d1a4392fdf05ba2ab1 (DEFAULT_TABLE table=test-table [id=c41d7ccd2ad74254a59d0bede959c66b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:08.797194 19925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c23463b0681945d1a4392fdf05ba2ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:08.797411 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.792245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42801 (local address 127.0.0.1:57900)
0620 14:50:08.792748 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:08.792754 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:08.792778 (+ 24us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:08.793087 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:08.793088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:08.793090 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:08.793123 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:08.793125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.794018 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.794019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:08.795454 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:08.795455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.797183 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.797184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:08.797229 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:08.797233 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:50:08.797279 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 14:50:08.797320 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":46,"thread_start_us":54,"threads_started":1}
I20260620 14:50:08.797694 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.792490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57900 (local address 127.0.98.2:42801)
0620 14:50:08.792878 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:08.792883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:08.792892 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:50:08.792926 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:08.792928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:08.792941 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:08.793006 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:08.793190 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.793929 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.796442 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.797029 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.797096 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:08.797154 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:08.797571 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 14:50:08.797620 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":75,"threads_started":1}
I20260620 14:50:08.798276 19791 tablet_service.cc:1467] Processing CreateTablet for tablet c23463b0681945d1a4392fdf05ba2ab1 (DEFAULT_TABLE table=test-table [id=c41d7ccd2ad74254a59d0bede959c66b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:08.798537 19791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c23463b0681945d1a4392fdf05ba2ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:08.799846 20005 tablet_bootstrap.cc:492] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04: Bootstrap starting.
I20260620 14:50:08.800069 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.792238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39065 (local address 127.0.0.1:60318)
0620 14:50:08.792410 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:08.792448 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:08.792506 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:08.792758 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:08.792761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:08.792763 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:08.792835 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:08.792837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.794065 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.794065 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:08.795449 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:08.795452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.796524 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.796525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:08.799897 (+ 3372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:08.799902 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:50:08.799960 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:50:08.800007 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":35,"thread_start_us":62,"threads_started":1}
I20260620 14:50:08.800367 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.792483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60318 (local address 127.0.98.1:39065)
0620 14:50:08.792629 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:08.792633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:08.792641 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:50:08.792670 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:08.792671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:08.792683 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:08.792742 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:08.792916 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.793982 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.795528 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.796286 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.796351 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:08.796407 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:08.800227 (+ 3820us) server_negotiation.cc:299] Negotiation successful
0620 14:50:08.800294 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260620 14:50:08.800590 20005 tablet_bootstrap.cc:654] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:08.800969 19657 tablet_service.cc:1467] Processing CreateTablet for tablet c23463b0681945d1a4392fdf05ba2ab1 (DEFAULT_TABLE table=test-table [id=c41d7ccd2ad74254a59d0bede959c66b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:08.801149 20005 log.cc:824] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:08.801287 19657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c23463b0681945d1a4392fdf05ba2ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:08.801326 20006 tablet_bootstrap.cc:492] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18: Bootstrap starting.
I20260620 14:50:08.801936 20006 tablet_bootstrap.cc:654] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:08.802032 20005 tablet_bootstrap.cc:492] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04: No bootstrap required, opened a new log
I20260620 14:50:08.802098 20005 ts_tablet_manager.cc:1397] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:50:08.802431 20006 log.cc:824] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:08.802999 20006 tablet_bootstrap.cc:492] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18: No bootstrap required, opened a new log
I20260620 14:50:08.803051 20006 ts_tablet_manager.cc:1397] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:50:08.803946 20005 raft_consensus.cc:348] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345c3386a2ff40008119a74cba28be0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39065 } } peers { permanent_uuid: "4df22153b6e0476a8163586faca56b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41117 } } peers { permanent_uuid: "fef701d7c3ea4700801deb826a832e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42801 } }
I20260620 14:50:08.804122 20005 raft_consensus.cc:374] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:08.804165 20005 raft_consensus.cc:729] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df22153b6e0476a8163586faca56b04, State: Initialized, Role: FOLLOWER
I20260620 14:50:08.804286 20005 consensus_queue.cc:260] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345c3386a2ff40008119a74cba28be0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39065 } } peers { permanent_uuid: "4df22153b6e0476a8163586faca56b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41117 } } peers { permanent_uuid: "fef701d7c3ea4700801deb826a832e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42801 } }
I20260620 14:50:08.804536 20005 ts_tablet_manager.cc:1428] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:50:08.804546 19991 heartbeater.cc:503] Master 127.0.98.62:43567 was elected leader, sending a full tablet report...
I20260620 14:50:08.804716 20009 tablet_bootstrap.cc:492] T c23463b0681945d1a4392fdf05ba2ab1 P 345c3386a2ff40008119a74cba28be0b: Bootstrap starting.
I20260620 14:50:08.805008 20006 raft_consensus.cc:348] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345c3386a2ff40008119a74cba28be0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39065 } } peers { permanent_uuid: "4df22153b6e0476a8163586faca56b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41117 } } peers { permanent_uuid: "fef701d7c3ea4700801deb826a832e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42801 } }
I20260620 14:50:08.805181 20006 raft_consensus.cc:374] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:08.805217 20006 raft_consensus.cc:729] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef701d7c3ea4700801deb826a832e18, State: Initialized, Role: FOLLOWER
I20260620 14:50:08.805222 20009 tablet_bootstrap.cc:654] T c23463b0681945d1a4392fdf05ba2ab1 P 345c3386a2ff40008119a74cba28be0b: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:08.805328 20006 consensus_queue.cc:260] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345c3386a2ff40008119a74cba28be0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39065 } } peers { permanent_uuid: "4df22153b6e0476a8163586faca56b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41117 } } peers { permanent_uuid: "fef701d7c3ea4700801deb826a832e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42801 } }
I20260620 14:50:08.805588 19857 heartbeater.cc:503] Master 127.0.98.62:43567 was elected leader, sending a full tablet report...
I20260620 14:50:08.805727 20009 log.cc:824] T c23463b0681945d1a4392fdf05ba2ab1 P 345c3386a2ff40008119a74cba28be0b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:08.805577 20006 ts_tablet_manager.cc:1428] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:50:08.806429 20009 tablet_bootstrap.cc:492] T c23463b0681945d1a4392fdf05ba2ab1 P 345c3386a2ff40008119a74cba28be0b: No bootstrap required, opened a new log
I20260620 14:50:08.806494 20009 ts_tablet_manager.cc:1397] T c23463b0681945d1a4392fdf05ba2ab1 P 345c3386a2ff40008119a74cba28be0b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:50:08.807889 20009 raft_consensus.cc:348] T c23463b0681945d1a4392fdf05ba2ab1 P 345c3386a2ff40008119a74cba28be0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345c3386a2ff40008119a74cba28be0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39065 } } peers { permanent_uuid: "4df22153b6e0476a8163586faca56b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41117 } } peers { permanent_uuid: "fef701d7c3ea4700801deb826a832e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42801 } }
I20260620 14:50:08.808012 20009 raft_consensus.cc:374] T c23463b0681945d1a4392fdf05ba2ab1 P 345c3386a2ff40008119a74cba28be0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:08.808037 20009 raft_consensus.cc:729] T c23463b0681945d1a4392fdf05ba2ab1 P 345c3386a2ff40008119a74cba28be0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345c3386a2ff40008119a74cba28be0b, State: Initialized, Role: FOLLOWER
I20260620 14:50:08.808141 20009 consensus_queue.cc:260] T c23463b0681945d1a4392fdf05ba2ab1 P 345c3386a2ff40008119a74cba28be0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345c3386a2ff40008119a74cba28be0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39065 } } peers { permanent_uuid: "4df22153b6e0476a8163586faca56b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41117 } } peers { permanent_uuid: "fef701d7c3ea4700801deb826a832e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42801 } }
I20260620 14:50:08.808395 20009 ts_tablet_manager.cc:1428] T c23463b0681945d1a4392fdf05ba2ab1 P 345c3386a2ff40008119a74cba28be0b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:50:08.808413 19723 heartbeater.cc:503] Master 127.0.98.62:43567 was elected leader, sending a full tablet report...
W20260620 14:50:08.868360 19858 tablet.cc:2369] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:50:08.881300 20010 raft_consensus.cc:482] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:50:08.881439 20010 raft_consensus.cc:504] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345c3386a2ff40008119a74cba28be0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39065 } } peers { permanent_uuid: "4df22153b6e0476a8163586faca56b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41117 } } peers { permanent_uuid: "fef701d7c3ea4700801deb826a832e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42801 } }
I20260620 14:50:08.881822 20010 leader_election.cc:288] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 345c3386a2ff40008119a74cba28be0b (127.0.98.1:39065), fef701d7c3ea4700801deb826a832e18 (127.0.98.2:42801)
I20260620 14:50:08.884811 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.881993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39065 (local address 127.0.98.3:54669)
0620 14:50:08.882047 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:08.882061 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:08.882122 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:08.882297 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:08.882299 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:08.882300 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:08.882387 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:08.882389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.883009 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.883010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:08.884033 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:08.884036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.884557 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.884558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:08.884637 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:08.884641 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:50:08.884693 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:50:08.884732 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 14:50:08.884811 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.882091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54669 (local address 127.0.98.1:39065)
0620 14:50:08.882136 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:08.882138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:08.882151 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:50:08.882183 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:08.882184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:08.882196 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:08.882284 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:08.882450 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.882942 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.884120 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.884480 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.884552 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:08.884563 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:08.884705 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 14:50:08.884746 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:50:08.885145 19677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23463b0681945d1a4392fdf05ba2ab1" candidate_uuid: "4df22153b6e0476a8163586faca56b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345c3386a2ff40008119a74cba28be0b" is_pre_election: true
I20260620 14:50:08.885262 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.882125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39259 (local address 127.0.98.2:42801)
0620 14:50:08.882175 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:08.882177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:08.882287 (+ 110us) server_negotiation.cc:407] Connection header received
0620 14:50:08.882875 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:08.882876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:08.882889 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:08.882951 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:08.883100 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.883605 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.884546 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.884936 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.885001 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:08.885011 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:08.885180 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 14:50:08.885210 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:50:08.885291 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.881994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42801 (local address 127.0.98.3:39259)
0620 14:50:08.882235 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:08.882243 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:08.882306 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:08.882961 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:08.882961 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:08.882963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:08.883034 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:08.883035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.883671 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.883672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:08.884464 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:08.884466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.885034 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.885035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:08.885110 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:08.885114 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:50:08.885172 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:50:08.885207 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":53,"thread_start_us":114,"threads_started":1}
I20260620 14:50:08.885331 19677 raft_consensus.cc:2454] T c23463b0681945d1a4392fdf05ba2ab1 P 345c3386a2ff40008119a74cba28be0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df22153b6e0476a8163586faca56b04 in term 0.
I20260620 14:50:08.885603 19811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23463b0681945d1a4392fdf05ba2ab1" candidate_uuid: "4df22153b6e0476a8163586faca56b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef701d7c3ea4700801deb826a832e18" is_pre_election: true
I20260620 14:50:08.885656 19880 leader_election.cc:302] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345c3386a2ff40008119a74cba28be0b, 4df22153b6e0476a8163586faca56b04; no voters:
I20260620 14:50:08.885736 19811 raft_consensus.cc:2454] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df22153b6e0476a8163586faca56b04 in term 0.
I20260620 14:50:08.885807 20010 raft_consensus.cc:2781] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:50:08.885874 20010 raft_consensus.cc:482] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:50:08.885895 20010 raft_consensus.cc:3037] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:08.886554 20010 raft_consensus.cc:504] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345c3386a2ff40008119a74cba28be0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39065 } } peers { permanent_uuid: "4df22153b6e0476a8163586faca56b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41117 } } peers { permanent_uuid: "fef701d7c3ea4700801deb826a832e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42801 } }
I20260620 14:50:08.886657 20010 leader_election.cc:288] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [CANDIDATE]: Term 1 election: Requested vote from peers 345c3386a2ff40008119a74cba28be0b (127.0.98.1:39065), fef701d7c3ea4700801deb826a832e18 (127.0.98.2:42801)
I20260620 14:50:08.886845 19811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23463b0681945d1a4392fdf05ba2ab1" candidate_uuid: "4df22153b6e0476a8163586faca56b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef701d7c3ea4700801deb826a832e18"
I20260620 14:50:08.886909 19811 raft_consensus.cc:3037] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:08.887485 19677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23463b0681945d1a4392fdf05ba2ab1" candidate_uuid: "4df22153b6e0476a8163586faca56b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345c3386a2ff40008119a74cba28be0b"
I20260620 14:50:08.887562 19677 raft_consensus.cc:3037] T c23463b0681945d1a4392fdf05ba2ab1 P 345c3386a2ff40008119a74cba28be0b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:08.887593 19811 raft_consensus.cc:2454] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df22153b6e0476a8163586faca56b04 in term 1.
I20260620 14:50:08.887742 19878 leader_election.cc:302] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df22153b6e0476a8163586faca56b04, fef701d7c3ea4700801deb826a832e18; no voters:
I20260620 14:50:08.887815 20010 raft_consensus.cc:2781] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:08.887944 20010 raft_consensus.cc:686] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [term 1 LEADER]: Becoming Leader. State: Replica: 4df22153b6e0476a8163586faca56b04, State: Running, Role: LEADER
I20260620 14:50:08.888033 20010 consensus_queue.cc:237] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345c3386a2ff40008119a74cba28be0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39065 } } peers { permanent_uuid: "4df22153b6e0476a8163586faca56b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41117 } } peers { permanent_uuid: "fef701d7c3ea4700801deb826a832e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42801 } }
I20260620 14:50:08.888439 19677 raft_consensus.cc:2454] T c23463b0681945d1a4392fdf05ba2ab1 P 345c3386a2ff40008119a74cba28be0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df22153b6e0476a8163586faca56b04 in term 1.
I20260620 14:50:08.888819 19532 catalog_manager.cc:5495] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4df22153b6e0476a8163586faca56b04 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4df22153b6e0476a8163586faca56b04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345c3386a2ff40008119a74cba28be0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4df22153b6e0476a8163586faca56b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef701d7c3ea4700801deb826a832e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42801 } health_report { overall_health: UNKNOWN } } }
I20260620 14:50:08.911247 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.908788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48552 (local address 127.0.98.3:41117)
0620 14:50:08.908862 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:08.908864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:08.908873 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:50:08.908906 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:08.908907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:08.908923 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:08.908984 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:08.909091 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.909550 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.910441 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:08.910699 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:08.910757 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:08.910938 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:08.911042 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:08.911114 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:08.911150 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 14:50:08.911170 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:50:08.912966 19677 raft_consensus.cc:1261] T c23463b0681945d1a4392fdf05ba2ab1 P 345c3386a2ff40008119a74cba28be0b [term 1 FOLLOWER]: Refusing update from remote peer 4df22153b6e0476a8163586faca56b04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:08.912966 19811 raft_consensus.cc:1261] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18 [term 1 FOLLOWER]: Refusing update from remote peer 4df22153b6e0476a8163586faca56b04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:08.913319 20015 consensus_queue.cc:1035] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef701d7c3ea4700801deb826a832e18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:08.913450 20010 consensus_queue.cc:1035] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "345c3386a2ff40008119a74cba28be0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:08.915570 20018 mvcc.cc:205] Tried to move back new op lower bound from 7298936868504866816 to 7298936868405964800. Current Snapshot: MvccSnapshot[applied={T|T < 7298936868504866816}]
I20260620 14:50:08.915840 20019 mvcc.cc:205] Tried to move back new op lower bound from 7298936868504866816 to 7298936868405964800. Current Snapshot: MvccSnapshot[applied={T|T < 7298936868504866816}]
I20260620 14:50:08.916399 20020 mvcc.cc:205] Tried to move back new op lower bound from 7298936868504866816 to 7298936868405964800. Current Snapshot: MvccSnapshot[applied={T|T < 7298936868504866816}]
I20260620 14:50:08.918274 19531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:08.918371 19531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:08.919102 19531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 955655352a6646cab0e3833d1c878cbf: Sending DeleteTablet for 3 replicas of tablet c23463b0681945d1a4392fdf05ba2ab1
I20260620 14:50:08.919401 19925 tablet_service.cc:1514] Processing DeleteTablet for tablet c23463b0681945d1a4392fdf05ba2ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48536
I20260620 14:50:08.919504 19657 tablet_service.cc:1514] Processing DeleteTablet for tablet c23463b0681945d1a4392fdf05ba2ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60318
I20260620 14:50:08.919507 19791 tablet_service.cc:1514] Processing DeleteTablet for tablet c23463b0681945d1a4392fdf05ba2ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57900
I20260620 14:50:08.919680 20028 tablet_replica.cc:330] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04: stopping tablet replica
I20260620 14:50:08.919775 20028 raft_consensus.cc:2229] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:50:08.919873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 19594
I20260620 14:50:08.919888 20028 raft_consensus.cc:2258] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:08.922283 20028 ts_tablet_manager.cc:1905] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:08.923542 20030 tablet_replica.cc:330] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18: stopping tablet replica
I20260620 14:50:08.923650 20030 raft_consensus.cc:2229] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:08.923756 20030 raft_consensus.cc:2258] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:08.923931 20028 ts_tablet_manager.cc:1918] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:08.924036 20028 log.cc:1196] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/c23463b0681945d1a4392fdf05ba2ab1
I20260620 14:50:08.924165 20030 ts_tablet_manager.cc:1905] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:08.924494 20028 ts_tablet_manager.cc:1939] T c23463b0681945d1a4392fdf05ba2ab1 P 4df22153b6e0476a8163586faca56b04: Deleting consensus metadata
I20260620 14:50:08.924944 19517 catalog_manager.cc:4841] TS 4df22153b6e0476a8163586faca56b04 (127.0.98.3:41117): tablet c23463b0681945d1a4392fdf05ba2ab1 (table test-table [id=c41d7ccd2ad74254a59d0bede959c66b]) successfully deleted
I20260620 14:50:08.925583 20030 ts_tablet_manager.cc:1918] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:08.925645 20030 log.cc:1196] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/c23463b0681945d1a4392fdf05ba2ab1
I20260620 14:50:08.925858 20030 ts_tablet_manager.cc:1939] T c23463b0681945d1a4392fdf05ba2ab1 P fef701d7c3ea4700801deb826a832e18: Deleting consensus metadata
I20260620 14:50:08.926249 19516 catalog_manager.cc:4841] TS fef701d7c3ea4700801deb826a832e18 (127.0.98.2:42801): tablet c23463b0681945d1a4392fdf05ba2ab1 (table test-table [id=c41d7ccd2ad74254a59d0bede959c66b]) successfully deleted
W20260620 14:50:08.927134 19518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39065 (error 108)
I20260620 14:50:08.927351 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 19728
I20260620 14:50:08.928086 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:08.927970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39065 (local address 127.0.0.1:60324)
0620 14:50:08.928019 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:08.928060 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 14:50:08.928256 19518 catalog_manager.cc:4584] TS 345c3386a2ff40008119a74cba28be0b (127.0.98.1:39065): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c23463b0681945d1a4392fdf05ba2ab1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39065: connect: Connection refused (error 111)
I20260620 14:50:08.932507 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 19862
I20260620 14:50:08.936992 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 19500
2026-06-20T14:50:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1831 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T14:50:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:08Z Disabled control of system clock
I20260620 14:50:08.962060 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44675
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:50:09.039798 20035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:09.040000 20035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:09.040019 20035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:09.041282 20035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:09.041329 20035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:09.041342 20035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:09.041352 20035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:09.041363 20035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:09.042701 20035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44675
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:09.042929 20035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:09.043205 20035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:09.047811 20040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:09.047791 20041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:09.047992 20035 server_base.cc:1034] running on GCE node
W20260620 14:50:09.047796 20043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:09.048275 20035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:09.048566 20035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:09.049803 20035 hybrid_clock.cc:648] HybridClock initialized: now 1781967009049790 us; error 33 us; skew 500 ppm
I20260620 14:50:09.051065 20035 webserver.cc:460] Webserver started at http://127.0.98.62:44023/ using document root <none> and password file <none>
I20260620 14:50:09.051307 20035 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:09.051357 20035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:09.051512 20035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:09.052568 20035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "f271f36adeca469ab94d42a4b4b571ed"
format_stamp: "Formatted at 2026-06-20 14:50:09 on dist-test-slave-cx89"
server_key: "d0e3b1705d97b83b0d9c6fab75e16256"
server_key_iv: "c16655cfa906d7f6aeb70357153e7294"
server_key_version: "encryptionkey@0"
I20260620 14:50:09.052922 20035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "f271f36adeca469ab94d42a4b4b571ed"
format_stamp: "Formatted at 2026-06-20 14:50:09 on dist-test-slave-cx89"
server_key: "d0e3b1705d97b83b0d9c6fab75e16256"
server_key_iv: "c16655cfa906d7f6aeb70357153e7294"
server_key_version: "encryptionkey@0"
I20260620 14:50:09.054170 20035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:50:09.054891 20049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:09.055055 20035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:09.055141 20035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "f271f36adeca469ab94d42a4b4b571ed"
format_stamp: "Formatted at 2026-06-20 14:50:09 on dist-test-slave-cx89"
server_key: "d0e3b1705d97b83b0d9c6fab75e16256"
server_key_iv: "c16655cfa906d7f6aeb70357153e7294"
server_key_version: "encryptionkey@0"
I20260620 14:50:09.055217 20035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:09.079397 20035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:09.079788 20035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:09.079903 20035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:09.083513 20035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44675
I20260620 14:50:09.083544 20101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44675 every 8 connection(s)
I20260620 14:50:09.083880 20035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:09.084878 20102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:09.086475 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 20035
I20260620 14:50:09.086589 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:09.086786 392 external_mini_cluster.cc:1428] Setting key fac99b5a77bd921127b645815fcb487c
I20260620 14:50:09.087380 20102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed: Bootstrap starting.
I20260620 14:50:09.087970 20102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:09.088248 20102 log.cc:824] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:09.088917 20102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed: No bootstrap required, opened a new log
I20260620 14:50:09.090968 20102 raft_consensus.cc:348] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f271f36adeca469ab94d42a4b4b571ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44675 } }
I20260620 14:50:09.091142 20102 raft_consensus.cc:374] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:09.091177 20102 raft_consensus.cc:729] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f271f36adeca469ab94d42a4b4b571ed, State: Initialized, Role: FOLLOWER
I20260620 14:50:09.091302 20102 consensus_queue.cc:260] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f271f36adeca469ab94d42a4b4b571ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44675 } }
I20260620 14:50:09.091365 20102 raft_consensus.cc:388] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:09.091403 20102 raft_consensus.cc:482] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:09.091460 20102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:09.092028 20102 raft_consensus.cc:504] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f271f36adeca469ab94d42a4b4b571ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44675 } }
I20260620 14:50:09.092124 20102 leader_election.cc:302] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f271f36adeca469ab94d42a4b4b571ed; no voters:
I20260620 14:50:09.092249 20102 leader_election.cc:288] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:09.092293 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.087364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.0.98.62:44675)
0620 14:50:09.087570 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:09.087578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:09.087603 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:50:09.088012 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:09.088018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:09.088090 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:09.088227 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:09.088453 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.089425 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.090471 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.090850 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.090927 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:09.091679 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:09.091682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:09.091682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:09.091833 (+ 151us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:09.091864 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:09.091873 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:09.091921 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:09.092011 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 14:50:09.092070 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260620 14:50:09.092312 20107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:09.092424 20107 raft_consensus.cc:686] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed [term 1 LEADER]: Becoming Leader. State: Replica: f271f36adeca469ab94d42a4b4b571ed, State: Running, Role: LEADER
I20260620 14:50:09.092479 20102 sys_catalog.cc:564] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:09.092499 20107 consensus_queue.cc:237] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f271f36adeca469ab94d42a4b4b571ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44675 } }
I20260620 14:50:09.092790 20107 sys_catalog.cc:455] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader f271f36adeca469ab94d42a4b4b571ed. Latest consensus state: current_term: 1 leader_uuid: "f271f36adeca469ab94d42a4b4b571ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f271f36adeca469ab94d42a4b4b571ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44675 } } }
I20260620 14:50:09.092870 20107 sys_catalog.cc:458] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:09.092800 20108 sys_catalog.cc:455] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f271f36adeca469ab94d42a4b4b571ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f271f36adeca469ab94d42a4b4b571ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44675 } } }
I20260620 14:50:09.093022 20108 sys_catalog.cc:458] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:09.093591 20113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:09.094015 20113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:09.095263 20113 catalog_manager.cc:1347] Generated new cluster ID: 9044373233c948ac96190004e084fab5
I20260620 14:50:09.095314 20113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:09.114817 20113 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:09.115335 20113 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:09.119911 20113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed: Generated new TSK 0
I20260620 14:50:09.120116 20113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:09.127566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44675
--builtin_ntp_servers=127.0.98.20:33005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:50:09.207290 20126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:09.207531 20126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:09.207556 20126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:09.208696 20126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:09.208735 20126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:09.208760 20126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:50:09.210084 20126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:09.210305 20126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:09.210585 20126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:09.215318 20131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:09.215312 20132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:09.215312 20126 server_base.cc:1034] running on GCE node
W20260620 14:50:09.215360 20134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:09.215708 20126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:09.215937 20126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:09.217069 20126 hybrid_clock.cc:648] HybridClock initialized: now 1781967009217056 us; error 29 us; skew 500 ppm
I20260620 14:50:09.218183 20126 webserver.cc:460] Webserver started at http://127.0.98.1:37573/ using document root <none> and password file <none>
I20260620 14:50:09.218403 20126 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:09.218452 20126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:09.218569 20126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:09.219617 20126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-0/data/instance:
uuid: "ff377488ee39497692806ef25856c664"
format_stamp: "Formatted at 2026-06-20 14:50:09 on dist-test-slave-cx89"
server_key: "bdac680bd3f08fec69d983875e1fbd81"
server_key_iv: "95d56441860ef8491005734107b8c0fa"
server_key_version: "encryptionkey@0"
I20260620 14:50:09.219933 20126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ff377488ee39497692806ef25856c664"
format_stamp: "Formatted at 2026-06-20 14:50:09 on dist-test-slave-cx89"
server_key: "bdac680bd3f08fec69d983875e1fbd81"
server_key_iv: "95d56441860ef8491005734107b8c0fa"
server_key_version: "encryptionkey@0"
I20260620 14:50:09.221112 20126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 14:50:09.221729 20140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:09.221871 20126 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:09.221949 20126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-0/wal
uuid: "ff377488ee39497692806ef25856c664"
format_stamp: "Formatted at 2026-06-20 14:50:09 on dist-test-slave-cx89"
server_key: "bdac680bd3f08fec69d983875e1fbd81"
server_key_iv: "95d56441860ef8491005734107b8c0fa"
server_key_version: "encryptionkey@0"
I20260620 14:50:09.222007 20126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:09.240438 20126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:09.240731 20126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:09.240854 20126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:09.241051 20126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:09.241348 20126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:09.241381 20126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:09.241408 20126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:09.241430 20126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:09.246853 20126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40415
I20260620 14:50:09.246892 20253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40415 every 8 connection(s)
I20260620 14:50:09.247190 20126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:50:09.252017 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.247897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42541 (local address 127.0.98.62:44675)
0620 14:50:09.247949 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:09.247951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:09.248169 (+ 218us) server_negotiation.cc:407] Connection header received
0620 14:50:09.248637 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:09.248638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:09.248652 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:09.248715 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:09.249353 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.249848 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.250764 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.250953 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.250995 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:09.251802 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:09.251803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:09.251804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:09.251808 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:09.251827 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:09.251831 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:09.251882 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:09.251935 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 14:50:09.251957 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:50:09.252233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 20126
I20260620 14:50:09.252305 20256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.247871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44675 (local address 127.0.98.1:42541)
0620 14:50:09.248054 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:09.248092 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:09.248551 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:09.248779 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:09.248782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:09.248790 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:09.249273 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:09.249278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.249921 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.249922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:09.250703 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:09.250705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.251006 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.251006 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:09.251112 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:09.251656 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:50:09.251656 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:09.251685 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:50:09.251707 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:09.251887 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:09.251889 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:09.251891 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:50:09.251937 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 14:50:09.252014 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260620 14:50:09.252324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:50:09.252537 392 external_mini_cluster.cc:1428] Setting key 97864221f9daa5c643f3a9ad743597ab
I20260620 14:50:09.252882 20254 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44675
I20260620 14:50:09.252977 20254 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:09.253212 20254 heartbeater.cc:511] Master 127.0.98.62:44675 requested a full tablet report, sending...
I20260620 14:50:09.253410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44675
--builtin_ntp_servers=127.0.98.20:33005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:50:09.253726 20066 ts_manager.cc:194] Registered new tserver with Master: ff377488ee39497692806ef25856c664 (127.0.98.1:40415)
I20260620 14:50:09.254565 20066 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:42541
W20260620 14:50:09.331962 20257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:09.332163 20257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:09.332191 20257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:09.333336 20257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:09.333375 20257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:09.333400 20257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:50:09.334703 20257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:09.334920 20257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:09.335199 20257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:09.339871 20262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:09.340008 20257 server_base.cc:1034] running on GCE node
W20260620 14:50:09.339850 20265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:09.339855 20263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:09.340267 20257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:09.340493 20257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:09.341648 20257 hybrid_clock.cc:648] HybridClock initialized: now 1781967009341632 us; error 35 us; skew 500 ppm
I20260620 14:50:09.342945 20257 webserver.cc:460] Webserver started at http://127.0.98.2:39851/ using document root <none> and password file <none>
I20260620 14:50:09.343176 20257 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:09.343225 20257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:09.343349 20257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:09.344487 20257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-1/data/instance:
uuid: "ad166d97f27443ab858f67418ded050b"
format_stamp: "Formatted at 2026-06-20 14:50:09 on dist-test-slave-cx89"
server_key: "c4759096a11ee423d902df1a9675162b"
server_key_iv: "91093267a031050ad1ce53747d4b1190"
server_key_version: "encryptionkey@0"
I20260620 14:50:09.344813 20257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ad166d97f27443ab858f67418ded050b"
format_stamp: "Formatted at 2026-06-20 14:50:09 on dist-test-slave-cx89"
server_key: "c4759096a11ee423d902df1a9675162b"
server_key_iv: "91093267a031050ad1ce53747d4b1190"
server_key_version: "encryptionkey@0"
I20260620 14:50:09.345956 20257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:50:09.346665 20271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:09.346817 20257 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:09.346896 20257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-1/wal
uuid: "ad166d97f27443ab858f67418ded050b"
format_stamp: "Formatted at 2026-06-20 14:50:09 on dist-test-slave-cx89"
server_key: "c4759096a11ee423d902df1a9675162b"
server_key_iv: "91093267a031050ad1ce53747d4b1190"
server_key_version: "encryptionkey@0"
I20260620 14:50:09.346961 20257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:09.354537 20257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:09.354809 20257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:09.354923 20257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:09.355101 20257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:09.355381 20257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:09.355432 20257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:09.355469 20257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:09.355494 20257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:09.360855 20257 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43659
I20260620 14:50:09.360914 20384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43659 every 8 connection(s)
I20260620 14:50:09.361189 20257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:50:09.365960 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.361861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46131 (local address 127.0.98.62:44675)
0620 14:50:09.361905 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:09.361907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:09.362076 (+ 169us) server_negotiation.cc:407] Connection header received
0620 14:50:09.362555 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:09.362556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:09.362564 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:09.362620 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:09.363217 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.363756 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.364739 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.364944 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.365005 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:09.365752 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:09.365753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:09.365753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:09.365757 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:09.365772 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:09.365776 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:09.365815 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:09.365878 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 14:50:09.365901 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:50:09.366144 20387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.361798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44675 (local address 127.0.98.2:46131)
0620 14:50:09.361973 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:09.362012 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:09.362476 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:09.362680 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:09.362683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:09.362691 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:09.363156 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:09.363160 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.363853 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.363854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:09.364621 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:09.364623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.365027 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.365028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:09.365136 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:09.365607 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:50:09.365608 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:09.365636 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:50:09.365671 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:09.365822 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:09.365825 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:09.365827 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:50:09.365872 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 14:50:09.365948 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260620 14:50:09.366622 20385 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44675
I20260620 14:50:09.366773 20385 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:09.366981 20385 heartbeater.cc:511] Master 127.0.98.62:44675 requested a full tablet report, sending...
I20260620 14:50:09.367303 20066 ts_manager.cc:194] Registered new tserver with Master: ad166d97f27443ab858f67418ded050b (127.0.98.2:43659)
I20260620 14:50:09.367702 20066 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46131
I20260620 14:50:09.368160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 20257
I20260620 14:50:09.368257 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:50:09.368414 392 external_mini_cluster.cc:1428] Setting key ee5fbabc8b34ce09f328f530bc5f3c01
I20260620 14:50:09.369204 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44675
--builtin_ntp_servers=127.0.98.20:33005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:50:09.447088 20388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:09.447286 20388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:09.447307 20388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:09.448433 20388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:09.448474 20388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:09.448499 20388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:50:09.449894 20388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:09.450109 20388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:09.450428 20388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:09.455109 20396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:09.455153 20394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:09.455277 20388 server_base.cc:1034] running on GCE node
W20260620 14:50:09.455330 20393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:09.455550 20388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:09.455796 20388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:09.456931 20388 hybrid_clock.cc:648] HybridClock initialized: now 1781967009456917 us; error 27 us; skew 500 ppm
I20260620 14:50:09.458035 20388 webserver.cc:460] Webserver started at http://127.0.98.3:39953/ using document root <none> and password file <none>
I20260620 14:50:09.458262 20388 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:09.458324 20388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:09.458444 20388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:09.459455 20388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-2/data/instance:
uuid: "8bc6fb06752a4a7a8eb48f53f068d72b"
format_stamp: "Formatted at 2026-06-20 14:50:09 on dist-test-slave-cx89"
server_key: "8652933780798f5c13c87319f3a1db25"
server_key_iv: "48bcf10a44169d67d2ecefb32b69305b"
server_key_version: "encryptionkey@0"
I20260620 14:50:09.459769 20388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8bc6fb06752a4a7a8eb48f53f068d72b"
format_stamp: "Formatted at 2026-06-20 14:50:09 on dist-test-slave-cx89"
server_key: "8652933780798f5c13c87319f3a1db25"
server_key_iv: "48bcf10a44169d67d2ecefb32b69305b"
server_key_version: "encryptionkey@0"
I20260620 14:50:09.460844 20388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:50:09.461432 20402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:09.461592 20388 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:50:09.461673 20388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-2/wal
uuid: "8bc6fb06752a4a7a8eb48f53f068d72b"
format_stamp: "Formatted at 2026-06-20 14:50:09 on dist-test-slave-cx89"
server_key: "8652933780798f5c13c87319f3a1db25"
server_key_iv: "48bcf10a44169d67d2ecefb32b69305b"
server_key_version: "encryptionkey@0"
I20260620 14:50:09.461740 20388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:09.476425 20388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:09.476699 20388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:09.476809 20388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:09.477015 20388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:09.477313 20388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:09.477344 20388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:09.477365 20388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:09.477377 20388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:09.482632 20388 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35095
I20260620 14:50:09.482699 20515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35095 every 8 connection(s)
I20260620 14:50:09.482956 20388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:50:09.483466 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 20388
I20260620 14:50:09.483573 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:50:09.483780 392 external_mini_cluster.cc:1428] Setting key ac78b91daa53a57639e25933d98bf10f
I20260620 14:50:09.487896 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.483989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49733 (local address 127.0.98.62:44675)
0620 14:50:09.484098 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:09.484099 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:09.484106 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:50:09.484598 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:09.484599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:09.484609 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:09.484670 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:09.485290 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.485780 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.486676 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.486863 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.486911 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:09.487690 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:09.487691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:09.487691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:09.487695 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:09.487709 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:09.487713 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:09.487766 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:09.487818 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 14:50:09.487839 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 14:50:09.488081 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.483675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44675 (local address 127.0.98.3:49733)
0620 14:50:09.483877 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:09.483921 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:09.484510 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:09.484726 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:09.484730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:09.484737 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:09.485212 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:09.485217 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.485846 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.485847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:09.486604 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:09.486606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.486933 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.486934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:09.487046 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:09.487542 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:50:09.487543 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:09.487570 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:50:09.487591 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:09.487769 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:09.487771 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:09.487774 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:50:09.487814 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 14:50:09.487887 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260620 14:50:09.488498 20516 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44675
I20260620 14:50:09.488576 20516 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:09.488746 20516 heartbeater.cc:511] Master 127.0.98.62:44675 requested a full tablet report, sending...
I20260620 14:50:09.489050 20066 ts_manager.cc:194] Registered new tserver with Master: 8bc6fb06752a4a7a8eb48f53f068d72b (127.0.98.3:35095)
I20260620 14:50:09.489368 20066 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:49733
I20260620 14:50:09.495299 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:50:09.499364 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.496548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.0.98.62:44675)
0620 14:50:09.496582 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:09.496584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:09.496689 (+ 105us) server_negotiation.cc:407] Connection header received
0620 14:50:09.496730 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:09.496731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:09.496738 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:09.496785 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:09.496920 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.497373 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.498195 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.498385 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.498439 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:09.499132 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:09.499133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:09.499133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:09.499137 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:09.499150 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:09.499153 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:09.499198 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:09.499235 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 14:50:09.499259 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 14:50:09.500880 20066 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:50:09.502431 20066 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:50:09.508338 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.504368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43659 (local address 127.0.0.1:57162)
0620 14:50:09.504696 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:09.504727 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:09.504815 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:09.505025 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:09.505028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:09.505030 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:09.505095 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:09.505097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.506014 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.506015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:09.507135 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:09.507137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.508044 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.508045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:09.508119 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:09.508133 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 14:50:09.508213 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:50:09.508262 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":91,"threads_started":1}
I20260620 14:50:09.508358 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.504457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57162 (local address 127.0.98.2:43659)
0620 14:50:09.504639 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:09.504644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:09.504786 (+ 142us) server_negotiation.cc:407] Connection header received
0620 14:50:09.504902 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:09.504906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:09.504921 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:09.505000 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:09.505165 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.505936 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.507255 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.507964 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.508039 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:09.508131 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:09.508233 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 14:50:09.508275 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
I20260620 14:50:09.508848 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.504554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35095 (local address 127.0.0.1:43264)
0620 14:50:09.504817 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:09.504826 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:09.504871 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:09.505098 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:09.505099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:09.505100 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:09.505127 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:09.505128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.506465 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.506466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:09.507470 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:09.507471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.508362 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.508363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:09.508687 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:09.508693 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:50:09.508746 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:50:09.508784 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":56,"threads_started":1}
I20260620 14:50:09.509160 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.504653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43264 (local address 127.0.98.3:35095)
0620 14:50:09.504796 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:09.504799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:09.504918 (+ 119us) server_negotiation.cc:407] Connection header received
0620 14:50:09.504960 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:09.504962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:09.504974 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:09.505041 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:09.505193 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.506382 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.507531 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.508191 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.508249 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:09.508327 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:09.509048 (+ 721us) server_negotiation.cc:299] Negotiation successful
0620 14:50:09.509087 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260620 14:50:09.509228 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.504453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58102 (local address 127.0.98.1:40415)
0620 14:50:09.504693 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:09.504698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:09.504866 (+ 168us) server_negotiation.cc:407] Connection header received
0620 14:50:09.504909 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:09.504911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:09.504924 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:09.504998 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:09.505279 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.506382 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.508311 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.508923 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.508984 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:09.509065 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:09.509119 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 14:50:09.509151 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
I20260620 14:50:09.509109 20319 tablet_service.cc:1467] Processing CreateTablet for tablet 9390ac321c5b491e88ae0b7c5ff4ba15 (DEFAULT_TABLE table=test-table [id=fac7843103b34e0b9e8bb7710517285b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:09.509217 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.504368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40415 (local address 127.0.0.1:58102)
0620 14:50:09.504666 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:09.504723 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:09.504794 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:09.505182 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:09.505183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:09.505184 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:09.505209 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:09.505210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.507468 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.507469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:09.508232 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:09.508234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.508998 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.508999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:09.509058 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:09.509062 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:50:09.509107 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 14:50:09.509143 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":56,"threads_started":1}
I20260620 14:50:09.509479 20319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9390ac321c5b491e88ae0b7c5ff4ba15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:09.510041 20188 tablet_service.cc:1467] Processing CreateTablet for tablet 9390ac321c5b491e88ae0b7c5ff4ba15 (DEFAULT_TABLE table=test-table [id=fac7843103b34e0b9e8bb7710517285b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:09.510334 20188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9390ac321c5b491e88ae0b7c5ff4ba15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:09.512140 20530 tablet_bootstrap.cc:492] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b: Bootstrap starting.
I20260620 14:50:09.512619 20450 tablet_service.cc:1467] Processing CreateTablet for tablet 9390ac321c5b491e88ae0b7c5ff4ba15 (DEFAULT_TABLE table=test-table [id=fac7843103b34e0b9e8bb7710517285b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:09.512827 20530 tablet_bootstrap.cc:654] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:09.512951 20450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9390ac321c5b491e88ae0b7c5ff4ba15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:09.513098 20531 tablet_bootstrap.cc:492] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ff377488ee39497692806ef25856c664: Bootstrap starting.
I20260620 14:50:09.513147 20530 log.cc:824] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:09.513692 20531 tablet_bootstrap.cc:654] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ff377488ee39497692806ef25856c664: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:09.513903 20530 tablet_bootstrap.cc:492] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b: No bootstrap required, opened a new log
I20260620 14:50:09.513963 20531 log.cc:824] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ff377488ee39497692806ef25856c664: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:09.513978 20530 ts_tablet_manager.cc:1397] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:50:09.514581 20531 tablet_bootstrap.cc:492] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ff377488ee39497692806ef25856c664: No bootstrap required, opened a new log
I20260620 14:50:09.514638 20531 ts_tablet_manager.cc:1397] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ff377488ee39497692806ef25856c664: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:50:09.516064 20534 tablet_bootstrap.cc:492] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b: Bootstrap starting.
I20260620 14:50:09.516489 20530 raft_consensus.cc:348] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad166d97f27443ab858f67418ded050b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43659 } } peers { permanent_uuid: "ff377488ee39497692806ef25856c664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40415 } } peers { permanent_uuid: "8bc6fb06752a4a7a8eb48f53f068d72b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35095 } }
I20260620 14:50:09.516619 20530 raft_consensus.cc:374] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:09.516671 20530 raft_consensus.cc:729] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad166d97f27443ab858f67418ded050b, State: Initialized, Role: FOLLOWER
I20260620 14:50:09.516758 20530 consensus_queue.cc:260] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad166d97f27443ab858f67418ded050b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43659 } } peers { permanent_uuid: "ff377488ee39497692806ef25856c664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40415 } } peers { permanent_uuid: "8bc6fb06752a4a7a8eb48f53f068d72b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35095 } }
I20260620 14:50:09.516999 20530 ts_tablet_manager.cc:1428] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:09.517004 20534 tablet_bootstrap.cc:654] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:09.517026 20385 heartbeater.cc:503] Master 127.0.98.62:44675 was elected leader, sending a full tablet report...
I20260620 14:50:09.517197 20531 raft_consensus.cc:348] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ff377488ee39497692806ef25856c664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad166d97f27443ab858f67418ded050b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43659 } } peers { permanent_uuid: "ff377488ee39497692806ef25856c664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40415 } } peers { permanent_uuid: "8bc6fb06752a4a7a8eb48f53f068d72b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35095 } }
I20260620 14:50:09.517357 20531 raft_consensus.cc:374] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ff377488ee39497692806ef25856c664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:09.517401 20531 raft_consensus.cc:729] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ff377488ee39497692806ef25856c664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff377488ee39497692806ef25856c664, State: Initialized, Role: FOLLOWER
I20260620 14:50:09.517424 20534 log.cc:824] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:09.517529 20531 consensus_queue.cc:260] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ff377488ee39497692806ef25856c664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad166d97f27443ab858f67418ded050b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43659 } } peers { permanent_uuid: "ff377488ee39497692806ef25856c664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40415 } } peers { permanent_uuid: "8bc6fb06752a4a7a8eb48f53f068d72b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35095 } }
I20260620 14:50:09.517762 20531 ts_tablet_manager.cc:1428] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ff377488ee39497692806ef25856c664: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:09.517782 20254 heartbeater.cc:503] Master 127.0.98.62:44675 was elected leader, sending a full tablet report...
I20260620 14:50:09.518254 20534 tablet_bootstrap.cc:492] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b: No bootstrap required, opened a new log
I20260620 14:50:09.518321 20534 ts_tablet_manager.cc:1397] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:50:09.520232 20534 raft_consensus.cc:348] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad166d97f27443ab858f67418ded050b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43659 } } peers { permanent_uuid: "ff377488ee39497692806ef25856c664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40415 } } peers { permanent_uuid: "8bc6fb06752a4a7a8eb48f53f068d72b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35095 } }
I20260620 14:50:09.520375 20534 raft_consensus.cc:374] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:09.520411 20534 raft_consensus.cc:729] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc6fb06752a4a7a8eb48f53f068d72b, State: Initialized, Role: FOLLOWER
I20260620 14:50:09.520493 20534 consensus_queue.cc:260] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad166d97f27443ab858f67418ded050b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43659 } } peers { permanent_uuid: "ff377488ee39497692806ef25856c664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40415 } } peers { permanent_uuid: "8bc6fb06752a4a7a8eb48f53f068d72b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35095 } }
I20260620 14:50:09.520669 20534 ts_tablet_manager.cc:1428] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:50:09.520730 20516 heartbeater.cc:503] Master 127.0.98.62:44675 was elected leader, sending a full tablet report...
I20260620 14:50:09.599900 20538 raft_consensus.cc:482] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:50:09.600018 20538 raft_consensus.cc:504] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad166d97f27443ab858f67418ded050b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43659 } } peers { permanent_uuid: "ff377488ee39497692806ef25856c664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40415 } } peers { permanent_uuid: "8bc6fb06752a4a7a8eb48f53f068d72b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35095 } }
I20260620 14:50:09.600383 20538 leader_election.cc:288] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad166d97f27443ab858f67418ded050b (127.0.98.2:43659), ff377488ee39497692806ef25856c664 (127.0.98.1:40415)
I20260620 14:50:09.603041 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.600607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46947 (local address 127.0.98.2:43659)
0620 14:50:09.600670 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:09.600688 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:09.600699 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:50:09.600734 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:09.600735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:09.600744 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:09.600800 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:09.600952 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.601443 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.602388 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.602747 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.602802 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:09.602812 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:09.602937 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:50:09.602972 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:50:09.603050 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.600697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35919 (local address 127.0.98.1:40415)
0620 14:50:09.600738 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:09.600740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:09.600766 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:50:09.600834 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:09.600835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:09.600846 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:09.600901 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:09.601035 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.601495 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.602406 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.602752 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.602811 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:09.602820 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:09.602952 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 14:50:09.602985 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:50:09.603050 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.600549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43659 (local address 127.0.98.3:46947)
0620 14:50:09.600624 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:09.600634 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:09.600680 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:09.600810 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:09.600810 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:09.600812 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:09.600890 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:09.600892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.601508 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.601509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:09.602298 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:09.602300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.602814 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.602815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:09.602881 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:09.602883 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:50:09.602921 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 14:50:09.602964 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":20}
I20260620 14:50:09.603070 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.600561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40415 (local address 127.0.98.3:35919)
0620 14:50:09.600712 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:09.600721 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:09.600780 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:09.600907 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:09.600908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:09.600909 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:09.600976 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:09.600977 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.601564 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.601565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:09.602314 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:09.602316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.602823 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.602823 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:09.602892 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:09.602895 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:50:09.602943 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:50:09.602978 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260620 14:50:09.603382 20339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9390ac321c5b491e88ae0b7c5ff4ba15" candidate_uuid: "8bc6fb06752a4a7a8eb48f53f068d72b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad166d97f27443ab858f67418ded050b" is_pre_election: true
I20260620 14:50:09.603554 20339 raft_consensus.cc:2454] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bc6fb06752a4a7a8eb48f53f068d72b in term 0.
I20260620 14:50:09.603561 20208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9390ac321c5b491e88ae0b7c5ff4ba15" candidate_uuid: "8bc6fb06752a4a7a8eb48f53f068d72b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff377488ee39497692806ef25856c664" is_pre_election: true
I20260620 14:50:09.603704 20208 raft_consensus.cc:2454] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ff377488ee39497692806ef25856c664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bc6fb06752a4a7a8eb48f53f068d72b in term 0.
I20260620 14:50:09.603765 20405 leader_election.cc:302] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bc6fb06752a4a7a8eb48f53f068d72b, ad166d97f27443ab858f67418ded050b; no voters:
I20260620 14:50:09.603878 20538 raft_consensus.cc:2781] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:50:09.603930 20538 raft_consensus.cc:482] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:50:09.603957 20538 raft_consensus.cc:3037] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:09.604584 20538 raft_consensus.cc:504] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad166d97f27443ab858f67418ded050b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43659 } } peers { permanent_uuid: "ff377488ee39497692806ef25856c664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40415 } } peers { permanent_uuid: "8bc6fb06752a4a7a8eb48f53f068d72b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35095 } }
I20260620 14:50:09.604768 20538 leader_election.cc:288] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [CANDIDATE]: Term 1 election: Requested vote from peers ad166d97f27443ab858f67418ded050b (127.0.98.2:43659), ff377488ee39497692806ef25856c664 (127.0.98.1:40415)
I20260620 14:50:09.604842 20208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9390ac321c5b491e88ae0b7c5ff4ba15" candidate_uuid: "8bc6fb06752a4a7a8eb48f53f068d72b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff377488ee39497692806ef25856c664"
I20260620 14:50:09.604882 20339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9390ac321c5b491e88ae0b7c5ff4ba15" candidate_uuid: "8bc6fb06752a4a7a8eb48f53f068d72b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad166d97f27443ab858f67418ded050b"
I20260620 14:50:09.604926 20208 raft_consensus.cc:3037] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ff377488ee39497692806ef25856c664 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:09.604950 20339 raft_consensus.cc:3037] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:09.605616 20339 raft_consensus.cc:2454] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bc6fb06752a4a7a8eb48f53f068d72b in term 1.
I20260620 14:50:09.605660 20208 raft_consensus.cc:2454] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ff377488ee39497692806ef25856c664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bc6fb06752a4a7a8eb48f53f068d72b in term 1.
I20260620 14:50:09.605772 20405 leader_election.cc:302] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bc6fb06752a4a7a8eb48f53f068d72b, ad166d97f27443ab858f67418ded050b; no voters:
I20260620 14:50:09.605865 20538 raft_consensus.cc:2781] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:09.605996 20538 raft_consensus.cc:686] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [term 1 LEADER]: Becoming Leader. State: Replica: 8bc6fb06752a4a7a8eb48f53f068d72b, State: Running, Role: LEADER
I20260620 14:50:09.606072 20538 consensus_queue.cc:237] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad166d97f27443ab858f67418ded050b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43659 } } peers { permanent_uuid: "ff377488ee39497692806ef25856c664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40415 } } peers { permanent_uuid: "8bc6fb06752a4a7a8eb48f53f068d72b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35095 } }
I20260620 14:50:09.606892 20065 catalog_manager.cc:5495] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bc6fb06752a4a7a8eb48f53f068d72b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8bc6fb06752a4a7a8eb48f53f068d72b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad166d97f27443ab858f67418ded050b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff377488ee39497692806ef25856c664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bc6fb06752a4a7a8eb48f53f068d72b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35095 } health_report { overall_health: HEALTHY } } }
W20260620 14:50:09.611624 20386 tablet.cc:2369] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:50:09.623081 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.620919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.0.98.3:35095)
0620 14:50:09.620970 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:09.620972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:09.620981 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:50:09.621048 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:09.621049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:09.621064 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:09.621123 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:09.621246 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.621706 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.622414 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:09.622593 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:09.622646 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:09.622784 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:09.622883 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:09.622953 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:09.622970 (+ 17us) server_negotiation.cc:299] Negotiation successful
0620 14:50:09.622990 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:50:09.624769 20339 raft_consensus.cc:1261] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b [term 1 FOLLOWER]: Refusing update from remote peer 8bc6fb06752a4a7a8eb48f53f068d72b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:09.624769 20208 raft_consensus.cc:1261] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ff377488ee39497692806ef25856c664 [term 1 FOLLOWER]: Refusing update from remote peer 8bc6fb06752a4a7a8eb48f53f068d72b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:09.625152 20540 consensus_queue.cc:1035] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff377488ee39497692806ef25856c664" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:09.625288 20538 consensus_queue.cc:1035] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad166d97f27443ab858f67418ded050b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:09.627444 20543 mvcc.cc:205] Tried to move back new op lower bound from 7298936871420792832 to 7298936871347204096. Current Snapshot: MvccSnapshot[applied={T|T < 7298936871420792832}]
I20260620 14:50:09.627835 20545 mvcc.cc:205] Tried to move back new op lower bound from 7298936871420792832 to 7298936871347204096. Current Snapshot: MvccSnapshot[applied={T|T < 7298936871420792832}]
I20260620 14:50:09.629469 20066 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:09.629546 20066 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:09.630196 20066 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f271f36adeca469ab94d42a4b4b571ed: Sending DeleteTablet for 3 replicas of tablet 9390ac321c5b491e88ae0b7c5ff4ba15
I20260620 14:50:09.630494 20319 tablet_service.cc:1514] Processing DeleteTablet for tablet 9390ac321c5b491e88ae0b7c5ff4ba15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:09 UTC) from {username='slave'} at 127.0.0.1:57162
I20260620 14:50:09.630524 20188 tablet_service.cc:1514] Processing DeleteTablet for tablet 9390ac321c5b491e88ae0b7c5ff4ba15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:09 UTC) from {username='slave'} at 127.0.0.1:58102
I20260620 14:50:09.630688 20553 tablet_replica.cc:330] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b: stopping tablet replica
I20260620 14:50:09.630754 20450 tablet_service.cc:1514] Processing DeleteTablet for tablet 9390ac321c5b491e88ae0b7c5ff4ba15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:09 UTC) from {username='slave'} at 127.0.0.1:43264
I20260620 14:50:09.630780 20553 raft_consensus.cc:2229] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:09.630842 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 20126
I20260620 14:50:09.630875 20553 raft_consensus.cc:2258] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:09.631839 20553 ts_tablet_manager.cc:1905] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:09.631899 20555 tablet_replica.cc:330] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b: stopping tablet replica
I20260620 14:50:09.631982 20555 raft_consensus.cc:2229] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:50:09.632089 20555 raft_consensus.cc:2258] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:09.632500 20555 ts_tablet_manager.cc:1905] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:09.633459 20553 ts_tablet_manager.cc:1918] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:09.633518 20553 log.cc:1196] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-1/wal/wals/9390ac321c5b491e88ae0b7c5ff4ba15
I20260620 14:50:09.633747 20553 ts_tablet_manager.cc:1939] T 9390ac321c5b491e88ae0b7c5ff4ba15 P ad166d97f27443ab858f67418ded050b: Deleting consensus metadata
I20260620 14:50:09.633941 20555 ts_tablet_manager.cc:1918] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:09.634004 20555 log.cc:1196] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b: Deleting WAL directory at /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966748446763-392-0/minicluster-data/ts-2/wal/wals/9390ac321c5b491e88ae0b7c5ff4ba15
I20260620 14:50:09.634150 20052 catalog_manager.cc:4841] TS ad166d97f27443ab858f67418ded050b (127.0.98.2:43659): tablet 9390ac321c5b491e88ae0b7c5ff4ba15 (table test-table [id=fac7843103b34e0b9e8bb7710517285b]) successfully deleted
I20260620 14:50:09.634212 20555 ts_tablet_manager.cc:1939] T 9390ac321c5b491e88ae0b7c5ff4ba15 P 8bc6fb06752a4a7a8eb48f53f068d72b: Deleting consensus metadata
I20260620 14:50:09.634548 20051 catalog_manager.cc:4841] TS 8bc6fb06752a4a7a8eb48f53f068d72b (127.0.98.3:35095): tablet 9390ac321c5b491e88ae0b7c5ff4ba15 (table test-table [id=fac7843103b34e0b9e8bb7710517285b]) successfully deleted
W20260620 14:50:09.636855 20051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40415 (error 108)
I20260620 14:50:09.637073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 20257
I20260620 14:50:09.637594 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:09.637159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40415 (local address 127.0.0.1:58118)
0620 14:50:09.637523 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:09.637564 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344}
W20260620 14:50:09.638841 20051 catalog_manager.cc:4584] TS ff377488ee39497692806ef25856c664 (127.0.98.1:40415): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9390ac321c5b491e88ae0b7c5ff4ba15: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40415: connect: Connection refused (error 111)
I20260620 14:50:09.641508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 20388
I20260620 14:50:09.645285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 20035
2026-06-20T14:50:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (707 ms)
[----------] 6 tests from EncryptionPolicyTest (7916 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:09 dist-test-slave-cx89 krb5kdc[20559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:09 dist-test-slave-cx89 krb5kdc[20559](info): set up 2 sockets
Jun 20 14:50:09 dist-test-slave-cx89 krb5kdc[20559](info): commencing operation
krb5kdc: starting...
W20260620 14:50:10.595075 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:10 dist-test-slave-cx89 krb5kdc[20559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:50:10.706349 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:10.784904 20572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:10.785106 20572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:10.785130 20572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:10.786224 20572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:10.786262 20572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:10.786274 20572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:10.786285 20572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:10.786295 20572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:10.787747 20572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:10.787927 20572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:10.788100 20572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:10.792548 20577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:10.792624 20572 server_base.cc:1034] running on GCE node
W20260620 14:50:10.792572 20580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:10.792572 20578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:10.793054 20572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:10.793325 20572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:10.794469 20572 hybrid_clock.cc:648] HybridClock initialized: now 1781967010794456 us; error 28 us; skew 500 ppm
Jun 20 14:50:10 dist-test-slave-cx89 krb5kdc[20559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:10.796829 20572 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:50:10.797252 20572 webserver.cc:460] Webserver started at http://127.0.0.1:45023/ using document root <none> and password file <none>
I20260620 14:50:10.797458 20572 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:10.797503 20572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:10.797622 20572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:10.798633 20572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "3ab59bb9074742dc96372fdeab414734"
format_stamp: "Formatted at 2026-06-20 14:50:10 on dist-test-slave-cx89"
server_key: "f48746d6a031271a0d9701ac373be664"
server_key_iv: "60d70a391e14e793fa6ea07cfe7a5532"
server_key_version: "encryptionkey@0"
I20260620 14:50:10.798951 20572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ab59bb9074742dc96372fdeab414734"
format_stamp: "Formatted at 2026-06-20 14:50:10 on dist-test-slave-cx89"
server_key: "f48746d6a031271a0d9701ac373be664"
server_key_iv: "60d70a391e14e793fa6ea07cfe7a5532"
server_key_version: "encryptionkey@0"
I20260620 14:50:10.800251 20572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:50:10.800977 20587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:10.801134 20572 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:10.801227 20572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "3ab59bb9074742dc96372fdeab414734"
format_stamp: "Formatted at 2026-06-20 14:50:10 on dist-test-slave-cx89"
server_key: "f48746d6a031271a0d9701ac373be664"
server_key_iv: "60d70a391e14e793fa6ea07cfe7a5532"
server_key_version: "encryptionkey@0"
I20260620 14:50:10.801292 20572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:10.823644 20572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:10.825928 20572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:10.826073 20572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:10.830192 20572 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34249
I20260620 14:50:10.830224 20639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34249 every 8 connection(s)
I20260620 14:50:10.830636 20572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:10.831535 20640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:10.834270 20640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734: Bootstrap starting.
I20260620 14:50:10.834822 20640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:10.835074 20640 log.cc:824] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:10.835724 20640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734: No bootstrap required, opened a new log
I20260620 14:50:10.837242 20640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab59bb9074742dc96372fdeab414734" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34249 } }
I20260620 14:50:10.837383 20640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:10.837416 20640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab59bb9074742dc96372fdeab414734, State: Initialized, Role: FOLLOWER
I20260620 14:50:10.837558 20640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab59bb9074742dc96372fdeab414734" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34249 } }
I20260620 14:50:10.837631 20640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:10.837663 20640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:10.837707 20640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:10.838186 20640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab59bb9074742dc96372fdeab414734" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34249 } }
I20260620 14:50:10.838269 20640 leader_election.cc:302] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ab59bb9074742dc96372fdeab414734; no voters:
I20260620 14:50:10.838413 20640 leader_election.cc:288] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:10.838459 20643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:10.838644 20643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734 [term 1 LEADER]: Becoming Leader. State: Replica: 3ab59bb9074742dc96372fdeab414734, State: Running, Role: LEADER
I20260620 14:50:10.838634 20640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:10.838764 20643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab59bb9074742dc96372fdeab414734" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34249 } }
I20260620 14:50:10.839100 20644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ab59bb9074742dc96372fdeab414734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab59bb9074742dc96372fdeab414734" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34249 } } }
I20260620 14:50:10.839193 20644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:10.839100 20645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ab59bb9074742dc96372fdeab414734. Latest consensus state: current_term: 1 leader_uuid: "3ab59bb9074742dc96372fdeab414734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab59bb9074742dc96372fdeab414734" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34249 } } }
I20260620 14:50:10.839318 20645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:10.839468 20647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:10.839958 20647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:10.840819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 20572
I20260620 14:50:10.840941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:10.841137 392 external_mini_cluster.cc:1428] Setting key dead6cfc8a1b0d3027bd2b861d11cc4e
I20260620 14:50:10.841496 20647 catalog_manager.cc:1347] Generated new cluster ID: ad1886f18d1d46bda6d5837453f210fd
I20260620 14:50:10.841549 20647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:50:10 dist-test-slave-cx89 krb5kdc[20559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:50:10.852105 20663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:10.841725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46451 (local address 127.0.0.1:34249)
0620 14:50:10.843820 (+ 2095us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:10.843827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:10.843850 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:50:10.843939 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:10.843944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:10.843984 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:10.844095 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:10.844837 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:10.845746 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:10.846470 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:10.846887 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:10.846977 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:10.849114 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:10.849119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:10.849119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:10.849130 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:10.850591 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:10.850938 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:10.850941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:10.850941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:10.851019 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:10.851212 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:10.851213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:10.851213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:10.851473 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:10.851630 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:10.851744 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 14:50:10.851936 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2054,"thread_start_us":74,"threads_started":1}
I20260620 14:50:10.852959 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:10.855046 20647 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:10.855584 20647 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:10.859615 20647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ab59bb9074742dc96372fdeab414734: Generated new TSK 0
I20260620 14:50:10.859814 20647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:10.866565 20663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:10.854139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55331 (local address 127.0.0.1:34249)
0620 14:50:10.854173 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:10.854175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:10.859504 (+ 5329us) server_negotiation.cc:407] Connection header received
0620 14:50:10.859531 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:10.859532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:10.859541 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:10.859660 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:10.860221 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:10.861283 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:10.861946 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:10.862289 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:10.862358 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:10.864348 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:10.864350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:10.864350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:10.864356 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:10.865552 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:10.865942 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:10.865943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:10.865944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:10.865991 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:10.866175 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:10.866175 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:10.866175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:10.866261 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:10.866329 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:10.866464 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 14:50:10.866507 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:50:10.868613 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 20572
W20260620 14:50:10.872471 20557 connection.cc:535] client connection to 127.0.0.1:34249 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34249): Connection reset by peer (error 104)
2026-06-20T14:50:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1222 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:10 dist-test-slave-cx89 krb5kdc[20671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:10 dist-test-slave-cx89 krb5kdc[20671](info): set up 2 sockets
Jun 20 14:50:10 dist-test-slave-cx89 krb5kdc[20671](info): commencing operation
krb5kdc: starting...
W20260620 14:50:11.811057 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:11 dist-test-slave-cx89 krb5kdc[20671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:50:11.919850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:11.998034 20684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:11.998230 20684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:11.998268 20684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:11.999382 20684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:11.999439 20684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:11.999460 20684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:11.999477 20684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:11.999500 20684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:12.000922 20684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46859
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:12.001151 20684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:12.001353 20684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:50:12.006340 20684 server_base.cc:1034] running on GCE node
W20260620 14:50:12.006290 20689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:12.006307 20690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:12.006314 20692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:12.006688 20684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:12.006973 20684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:12.008143 20684 hybrid_clock.cc:648] HybridClock initialized: now 1781967012008123 us; error 38 us; skew 500 ppm
Jun 20 14:50:12 dist-test-slave-cx89 krb5kdc[20671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:12.014514 20684 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:50:12.014928 20684 webserver.cc:460] Webserver started at http://127.0.0.1:40825/ using document root <none> and password file <none>
I20260620 14:50:12.015120 20684 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:12.015168 20684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:12.015295 20684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:12.016399 20684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "93545457e94b4fd6870171c7eb885fcb"
format_stamp: "Formatted at 2026-06-20 14:50:12 on dist-test-slave-cx89"
server_key: "ab9c7bfdee7245cd8980e26b6fb6fa7f"
server_key_iv: "e37dd888b5827b696959908149aee5e0"
server_key_version: "encryptionkey@0"
I20260620 14:50:12.016731 20684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "93545457e94b4fd6870171c7eb885fcb"
format_stamp: "Formatted at 2026-06-20 14:50:12 on dist-test-slave-cx89"
server_key: "ab9c7bfdee7245cd8980e26b6fb6fa7f"
server_key_iv: "e37dd888b5827b696959908149aee5e0"
server_key_version: "encryptionkey@0"
I20260620 14:50:12.017951 20684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:50:12.018570 20699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:12.018730 20684 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:12.018808 20684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "93545457e94b4fd6870171c7eb885fcb"
format_stamp: "Formatted at 2026-06-20 14:50:12 on dist-test-slave-cx89"
server_key: "ab9c7bfdee7245cd8980e26b6fb6fa7f"
server_key_iv: "e37dd888b5827b696959908149aee5e0"
server_key_version: "encryptionkey@0"
I20260620 14:50:12.018873 20684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:12.028053 20684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:12.029844 20684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:12.029969 20684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:12.033488 20684 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46859
I20260620 14:50:12.033530 20751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46859 every 8 connection(s)
I20260620 14:50:12.033821 20684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:12.033952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 20684
I20260620 14:50:12.034047 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:12.034251 392 external_mini_cluster.cc:1428] Setting key 81b651d7c4586fe7a3aac841459cd055
I20260620 14:50:12.034708 20752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:12.037703 20752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb: Bootstrap starting.
I20260620 14:50:12.038321 20752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:12.038607 20752 log.cc:824] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:12.039345 20752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb: No bootstrap required, opened a new log
I20260620 14:50:12.039819 20755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:12.034727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48295 (local address 127.0.0.1:46859)
0620 14:50:12.034904 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:12.034913 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:12.034941 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:50:12.035430 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:12.035440 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:12.035499 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:12.035650 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:12.035808 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:12.037116 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:12.037826 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:12.038394 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:12.038505 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:12.039131 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:12.039136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:12.039136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:12.039325 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:12.039361 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:12.039376 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:12.039491 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:12.039561 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:50:12.039586 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260620 14:50:12.040729 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:12.041441 20752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93545457e94b4fd6870171c7eb885fcb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46859 } }
I20260620 14:50:12.041603 20752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:12.041642 20752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93545457e94b4fd6870171c7eb885fcb, State: Initialized, Role: FOLLOWER
I20260620 14:50:12.041762 20752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93545457e94b4fd6870171c7eb885fcb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46859 } }
I20260620 14:50:12.041848 20752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:12.041888 20752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:12.041934 20752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:12.042586 20752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93545457e94b4fd6870171c7eb885fcb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46859 } }
I20260620 14:50:12.042683 20752 leader_election.cc:302] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93545457e94b4fd6870171c7eb885fcb; no voters:
I20260620 14:50:12.042840 20752 leader_election.cc:288] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:12.042868 20762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:12.043015 20762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb [term 1 LEADER]: Becoming Leader. State: Replica: 93545457e94b4fd6870171c7eb885fcb, State: Running, Role: LEADER
I20260620 14:50:12.043069 20752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:12.043128 20762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93545457e94b4fd6870171c7eb885fcb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46859 } }
I20260620 14:50:12.043449 20762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93545457e94b4fd6870171c7eb885fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93545457e94b4fd6870171c7eb885fcb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46859 } } }
I20260620 14:50:12.043586 20762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:12.043754 20762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93545457e94b4fd6870171c7eb885fcb. Latest consensus state: current_term: 1 leader_uuid: "93545457e94b4fd6870171c7eb885fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93545457e94b4fd6870171c7eb885fcb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46859 } } }
I20260620 14:50:12.043855 20762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:12.044134 20771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:12.044627 20771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:12.045265 20755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:12.042171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34007 (local address 127.0.0.1:46859)
0620 14:50:12.042208 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:12.042210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:12.042310 (+ 100us) server_negotiation.cc:407] Connection header received
0620 14:50:12.042375 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:12.042376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:12.042383 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:12.042434 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:12.042574 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:12.043159 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:12.044119 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:12.044412 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:12.044473 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:12.045048 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:12.045049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:12.045050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:12.045054 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:12.045067 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:12.045072 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:12.045119 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:12.045184 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 14:50:12.045198 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:50:12.046077 20771 catalog_manager.cc:1347] Generated new cluster ID: 5d62c59f02c245e4ab048655ae46c461
I20260620 14:50:12.046128 20771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:12.061214 20771 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:12.061760 20771 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:12.066672 20771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93545457e94b4fd6870171c7eb885fcb: Generated new TSK 0
I20260620 14:50:12.066839 20771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:12.072679 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 20684
W20260620 14:50:12.076299 20669 connection.cc:535] client connection to 127.0.0.1:46859 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46859): Connection reset by peer (error 104)
2026-06-20T14:50:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1203 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:12 dist-test-slave-cx89 krb5kdc[20783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:12 dist-test-slave-cx89 krb5kdc[20783](info): set up 2 sockets
Jun 20 14:50:12 dist-test-slave-cx89 krb5kdc[20783](info): commencing operation
krb5kdc: starting...
W20260620 14:50:13.027122 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:13 dist-test-slave-cx89 krb5kdc[20783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:50:13.136400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36645
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:13.218034 20796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:13.218235 20796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:13.218256 20796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:13.219336 20796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:13.219372 20796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:13.219385 20796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:13.219395 20796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:13.219406 20796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:13.220852 20796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36645
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:13.221037 20796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:13.221203 20796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:13.225845 20801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:13.225843 20804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:13.225838 20802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:13.225868 20796 server_base.cc:1034] running on GCE node
I20260620 14:50:13.226320 20796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:13.226584 20796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:13.227726 20796 hybrid_clock.cc:648] HybridClock initialized: now 1781967013227713 us; error 29 us; skew 500 ppm
Jun 20 14:50:13 dist-test-slave-cx89 krb5kdc[20783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:13.230234 20796 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:50:13.230669 20796 webserver.cc:460] Webserver started at http://127.0.0.1:36403/ using document root <none> and password file <none>
I20260620 14:50:13.230867 20796 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:13.230914 20796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:13.231034 20796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:13.232204 20796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "7585654897f7499bae0b80fff00d2d7e"
format_stamp: "Formatted at 2026-06-20 14:50:13 on dist-test-slave-cx89"
server_key: "10beccffb912abda9f607e165c221d1f"
server_key_iv: "d3c75b1e5d9b144077444e4d2f7d6659"
server_key_version: "encryptionkey@0"
I20260620 14:50:13.232530 20796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "7585654897f7499bae0b80fff00d2d7e"
format_stamp: "Formatted at 2026-06-20 14:50:13 on dist-test-slave-cx89"
server_key: "10beccffb912abda9f607e165c221d1f"
server_key_iv: "d3c75b1e5d9b144077444e4d2f7d6659"
server_key_version: "encryptionkey@0"
I20260620 14:50:13.233760 20796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:50:13.234462 20811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:13.234608 20796 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:50:13.234688 20796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "7585654897f7499bae0b80fff00d2d7e"
format_stamp: "Formatted at 2026-06-20 14:50:13 on dist-test-slave-cx89"
server_key: "10beccffb912abda9f607e165c221d1f"
server_key_iv: "d3c75b1e5d9b144077444e4d2f7d6659"
server_key_version: "encryptionkey@0"
I20260620 14:50:13.234753 20796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:13.237901 20796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:13.239569 20796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:13.239693 20796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:13.243240 20796 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36645
I20260620 14:50:13.243269 20863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36645 every 8 connection(s)
I20260620 14:50:13.243590 20796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:13.244469 20864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:13.246603 20864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e: Bootstrap starting.
I20260620 14:50:13.247153 20864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:13.247403 20864 log.cc:824] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:13.248060 20864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e: No bootstrap required, opened a new log
I20260620 14:50:13.249588 20864 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7585654897f7499bae0b80fff00d2d7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36645 } }
I20260620 14:50:13.249722 20864 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:13.249744 20864 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7585654897f7499bae0b80fff00d2d7e, State: Initialized, Role: FOLLOWER
I20260620 14:50:13.249877 20864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7585654897f7499bae0b80fff00d2d7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36645 } }
I20260620 14:50:13.249948 20864 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:13.249979 20864 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:13.250026 20864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:13.250530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 20796
I20260620 14:50:13.250574 20864 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7585654897f7499bae0b80fff00d2d7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36645 } }
I20260620 14:50:13.250667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:13.250667 20864 leader_election.cc:302] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7585654897f7499bae0b80fff00d2d7e; no voters:
I20260620 14:50:13.250813 20864 leader_election.cc:288] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:13.250867 392 external_mini_cluster.cc:1428] Setting key 3a94e6d5933881f0b54a543c76083735
I20260620 14:50:13.250856 20867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:13.251063 20864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:13.251187 20867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e [term 1 LEADER]: Becoming Leader. State: Replica: 7585654897f7499bae0b80fff00d2d7e, State: Running, Role: LEADER
I20260620 14:50:13.251288 20867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7585654897f7499bae0b80fff00d2d7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36645 } }
I20260620 14:50:13.251633 20868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7585654897f7499bae0b80fff00d2d7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7585654897f7499bae0b80fff00d2d7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36645 } } }
I20260620 14:50:13.251711 20868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:13.251868 20868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7585654897f7499bae0b80fff00d2d7e. Latest consensus state: current_term: 1 leader_uuid: "7585654897f7499bae0b80fff00d2d7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7585654897f7499bae0b80fff00d2d7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36645 } } }
I20260620 14:50:13.251948 20878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:13.251971 20868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:13.252900 20878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:13.253364 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:13.251534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56487 (local address 127.0.0.1:36645)
0620 14:50:13.251708 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:13.251717 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:13.251750 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:50:13.251867 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:13.251874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:13.251905 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:13.252307 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:13.252768 (+ 461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:13.252772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:13.252772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:13.252776 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:13.252815 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:13.252829 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:13.252878 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:13.252962 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 14:50:13.253086 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":60,"threads_started":1}
I20260620 14:50:13.254191 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:13.254328 20878 catalog_manager.cc:1347] Generated new cluster ID: 715c10d6cb68418eb12cf67f97ca40fe
I20260620 14:50:13.254365 20878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:13.264550 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:13.255603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35551 (local address 127.0.0.1:36645)
0620 14:50:13.255636 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:13.255638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:13.263592 (+ 7954us) server_negotiation.cc:407] Connection header received
0620 14:50:13.263658 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:13.263659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:13.263665 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:13.263720 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:13.264339 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:13.264340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:13.264341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:13.264345 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:13.264361 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:13.264365 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:13.264421 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:13.264456 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 14:50:13.264470 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:50:13.276515 20878 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:13.276698 20878 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:13.280853 20878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7585654897f7499bae0b80fff00d2d7e: Generated new TSK 0
I20260620 14:50:13.281037 20878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:13.284863 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 20796
2026-06-20T14:50:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1212 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:13 dist-test-slave-cx89 krb5kdc[20895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:13 dist-test-slave-cx89 krb5kdc[20895](info): set up 2 sockets
Jun 20 14:50:13 dist-test-slave-cx89 krb5kdc[20895](info): commencing operation
krb5kdc: starting...
W20260620 14:50:14.239078 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:14 dist-test-slave-cx89 krb5kdc[20895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:50:14.350063 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40285
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:14.428215 20908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:14.428428 20908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:14.428460 20908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:14.429603 20908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:14.429646 20908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:14.429664 20908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:14.429679 20908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:14.429695 20908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:14.431138 20908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:14.431370 20908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:14.431594 20908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:14.436022 20913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:14.436007 20914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:14.436009 20916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:14.436118 20908 server_base.cc:1034] running on GCE node
I20260620 14:50:14.436401 20908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:14.436666 20908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:14.437803 20908 hybrid_clock.cc:648] HybridClock initialized: now 1781967014437787 us; error 34 us; skew 500 ppm
Jun 20 14:50:14 dist-test-slave-cx89 krb5kdc[20895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:14.440109 20908 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:50:14.440495 20908 webserver.cc:460] Webserver started at http://127.0.0.1:37361/ using document root <none> and password file <none>
I20260620 14:50:14.440681 20908 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:14.440727 20908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:14.440848 20908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:14.441879 20908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "9d8d04eb23f541a48a01714269e9d87c"
format_stamp: "Formatted at 2026-06-20 14:50:14 on dist-test-slave-cx89"
server_key: "42a34164696463678fdc49c69b5d1774"
server_key_iv: "7bab1e40be7480431f1a45ff9b5a0fa3"
server_key_version: "encryptionkey@0"
I20260620 14:50:14.442211 20908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "9d8d04eb23f541a48a01714269e9d87c"
format_stamp: "Formatted at 2026-06-20 14:50:14 on dist-test-slave-cx89"
server_key: "42a34164696463678fdc49c69b5d1774"
server_key_iv: "7bab1e40be7480431f1a45ff9b5a0fa3"
server_key_version: "encryptionkey@0"
I20260620 14:50:14.443403 20908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:50:14.444111 20923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:14.444258 20908 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:50:14.444339 20908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "9d8d04eb23f541a48a01714269e9d87c"
format_stamp: "Formatted at 2026-06-20 14:50:14 on dist-test-slave-cx89"
server_key: "42a34164696463678fdc49c69b5d1774"
server_key_iv: "7bab1e40be7480431f1a45ff9b5a0fa3"
server_key_version: "encryptionkey@0"
I20260620 14:50:14.444398 20908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:14.465575 20908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:14.467747 20908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:14.467883 20908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:14.471448 20908 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40285
I20260620 14:50:14.471484 20975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40285 every 8 connection(s)
I20260620 14:50:14.471791 20908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:14.472570 20976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:14.474293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 20908
I20260620 14:50:14.474400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:14.474565 392 external_mini_cluster.cc:1428] Setting key 68896b4e434e494da5f663ecb1773d5e
I20260620 14:50:14.474821 20976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c: Bootstrap starting.
I20260620 14:50:14.475453 20976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:14.475719 20976 log.cc:824] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:14.476416 20976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c: No bootstrap required, opened a new log
I20260620 14:50:14.478374 20976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8d04eb23f541a48a01714269e9d87c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40285 } }
I20260620 14:50:14.478542 20976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:14.478569 20976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d8d04eb23f541a48a01714269e9d87c, State: Initialized, Role: FOLLOWER
I20260620 14:50:14.478686 20976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8d04eb23f541a48a01714269e9d87c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40285 } }
I20260620 14:50:14.478749 20976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:14.478770 20976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:14.478812 20976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:14.479451 20976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8d04eb23f541a48a01714269e9d87c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40285 } }
I20260620 14:50:14.479561 20976 leader_election.cc:302] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d8d04eb23f541a48a01714269e9d87c; no voters:
I20260620 14:50:14.479718 20976 leader_election.cc:288] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:14.479758 20981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:14.479894 20981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c [term 1 LEADER]: Becoming Leader. State: Replica: 9d8d04eb23f541a48a01714269e9d87c, State: Running, Role: LEADER
I20260620 14:50:14.479977 20976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:14.480039 20981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8d04eb23f541a48a01714269e9d87c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40285 } }
I20260620 14:50:14.480365 20981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d8d04eb23f541a48a01714269e9d87c. Latest consensus state: current_term: 1 leader_uuid: "9d8d04eb23f541a48a01714269e9d87c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8d04eb23f541a48a01714269e9d87c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40285 } } }
I20260620 14:50:14.480571 20981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:14.480510 20983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d8d04eb23f541a48a01714269e9d87c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8d04eb23f541a48a01714269e9d87c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40285 } } }
I20260620 14:50:14.480929 20983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:14.481110 20990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 14:50:14 dist-test-slave-cx89 krb5kdc[20895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:50:14.481595 20990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:14.485140 20979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:14.474987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50831 (local address 127.0.0.1:40285)
0620 14:50:14.475157 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:14.475165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:14.475190 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:50:14.475665 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:14.475673 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:14.475722 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:14.475844 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:14.476576 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:14.477413 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:14.478444 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:14.478962 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:14.479041 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:14.481736 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:14.481740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:14.481741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:14.481752 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:14.483608 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:14.483936 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:14.483939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:14.483939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:14.484043 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:14.484207 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:14.484208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:14.484208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:14.484515 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:14.484686 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:14.484789 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 14:50:14.484927 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260620 14:50:14.486047 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:14.487828 20990 catalog_manager.cc:1347] Generated new cluster ID: e76be1c68d8942b4b27863f59bd346ca
I20260620 14:50:14.487881 20990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:14.493112 20979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:14.487256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42341 (local address 127.0.0.1:40285)
0620 14:50:14.487298 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:14.487299 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:14.487907 (+ 608us) server_negotiation.cc:407] Connection header received
0620 14:50:14.488001 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:14.488002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:14.488011 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:14.488053 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:14.488512 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:14.489006 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:14.489645 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:14.489865 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:14.489915 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:14.490978 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:14.490980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:14.490980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:14.490994 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:14.492198 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:14.492477 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:14.492478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:14.492478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:14.492529 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:14.492681 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:14.492681 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:14.492682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:14.492780 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:14.492879 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:14.492960 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 14:50:14.493032 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:50:14.497989 20990 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:14.498653 20990 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:14.503448 20990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d8d04eb23f541a48a01714269e9d87c: Generated new TSK 0
I20260620 14:50:14.503612 20990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:14.511443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 20908
2026-06-20T14:50:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1226 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:14 dist-test-slave-cx89 krb5kdc[21007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:14 dist-test-slave-cx89 krb5kdc[21007](info): set up 2 sockets
Jun 20 14:50:14 dist-test-slave-cx89 krb5kdc[21007](info): commencing operation
krb5kdc: starting...
W20260620 14:50:15.463129 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:15 dist-test-slave-cx89 krb5kdc[21007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:50:15.570078 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:15.650513 21020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:15.650722 21020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:15.650751 21020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:15.651921 21020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:15.651963 21020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:15.651981 21020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:15.651996 21020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:15.652014 21020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:15.653463 21020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:15.653702 21020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:15.653893 21020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:15.658407 21025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:15.658407 21026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:15.658433 21020 server_base.cc:1034] running on GCE node
W20260620 14:50:15.658464 21028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:15.658862 21020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:15.659139 21020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:15.660275 21020 hybrid_clock.cc:648] HybridClock initialized: now 1781967015660261 us; error 26 us; skew 500 ppm
Jun 20 14:50:15 dist-test-slave-cx89 krb5kdc[21007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:15.662699 21020 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:50:15.663115 21020 webserver.cc:460] Webserver started at http://127.0.0.1:39141/ using document root <none> and password file <none>
I20260620 14:50:15.663304 21020 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:15.663352 21020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:15.663497 21020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:15.664520 21020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "58d07d3691bd400991e24e4f322e8ffb"
format_stamp: "Formatted at 2026-06-20 14:50:15 on dist-test-slave-cx89"
server_key: "0037bfe5f86a0cb70102d9132e16fb5d"
server_key_iv: "9cf191af584a9c7c5b8e2ff31a60b850"
server_key_version: "encryptionkey@0"
I20260620 14:50:15.664840 21020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "58d07d3691bd400991e24e4f322e8ffb"
format_stamp: "Formatted at 2026-06-20 14:50:15 on dist-test-slave-cx89"
server_key: "0037bfe5f86a0cb70102d9132e16fb5d"
server_key_iv: "9cf191af584a9c7c5b8e2ff31a60b850"
server_key_version: "encryptionkey@0"
I20260620 14:50:15.665978 21020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:50:15.666574 21035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:15.666718 21020 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:15.666795 21020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "58d07d3691bd400991e24e4f322e8ffb"
format_stamp: "Formatted at 2026-06-20 14:50:15 on dist-test-slave-cx89"
server_key: "0037bfe5f86a0cb70102d9132e16fb5d"
server_key_iv: "9cf191af584a9c7c5b8e2ff31a60b850"
server_key_version: "encryptionkey@0"
I20260620 14:50:15.666858 21020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:15.682938 21020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:15.685168 21020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:15.685321 21020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:15.689284 21020 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42973
I20260620 14:50:15.689311 21087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42973 every 8 connection(s)
I20260620 14:50:15.689656 21020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:15.690474 21088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:15.692736 21088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb: Bootstrap starting.
I20260620 14:50:15.693260 21088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:15.693529 21088 log.cc:824] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:15.694233 21088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb: No bootstrap required, opened a new log
I20260620 14:50:15.694413 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 21020
I20260620 14:50:15.694519 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:15.694681 392 external_mini_cluster.cc:1428] Setting key 2a1d95cfd240269d2b28f339043cd177
I20260620 14:50:15.696292 21088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d07d3691bd400991e24e4f322e8ffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42973 } }
I20260620 14:50:15.696466 21088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:15.696499 21088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d07d3691bd400991e24e4f322e8ffb, State: Initialized, Role: FOLLOWER
I20260620 14:50:15.696621 21088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d07d3691bd400991e24e4f322e8ffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42973 } }
I20260620 14:50:15.696691 21088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:15.696719 21088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:15.696763 21088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:15.697245 21088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d07d3691bd400991e24e4f322e8ffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42973 } }
I20260620 14:50:15.697350 21088 leader_election.cc:302] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58d07d3691bd400991e24e4f322e8ffb; no voters:
I20260620 14:50:15.697497 21088 leader_election.cc:288] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:15.697527 21093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:15.697623 21093 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb [term 1 LEADER]: Becoming Leader. State: Replica: 58d07d3691bd400991e24e4f322e8ffb, State: Running, Role: LEADER
I20260620 14:50:15.697701 21093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d07d3691bd400991e24e4f322e8ffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42973 } }
I20260620 14:50:15.697733 21088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:15.697994 21093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58d07d3691bd400991e24e4f322e8ffb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d07d3691bd400991e24e4f322e8ffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42973 } } }
I20260620 14:50:15.698117 21093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:15.698275 21093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58d07d3691bd400991e24e4f322e8ffb. Latest consensus state: current_term: 1 leader_uuid: "58d07d3691bd400991e24e4f322e8ffb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d07d3691bd400991e24e4f322e8ffb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42973 } } }
I20260620 14:50:15.698375 21093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:15.698678 21100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:15.699162 21100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:15.700677 21092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:15.695129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52991 (local address 127.0.0.1:42973)
0620 14:50:15.695282 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:15.695290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:15.695316 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:50:15.695913 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:15.695920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:15.695975 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:15.696086 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:15.696317 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:15.697538 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:15.698535 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:15.699133 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:15.699229 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:15.700040 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:15.700044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:15.700044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:15.700225 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:15.700263 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:15.700279 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:15.700346 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:15.700407 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 14:50:15.700466 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20260620 14:50:15.700770 21100 catalog_manager.cc:1347] Generated new cluster ID: c9af6d1c8b224f3dbc526dcda9bad48c
I20260620 14:50:15.700852 21100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:15.701489 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:15.705178 21092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:15.702763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33369 (local address 127.0.0.1:42973)
0620 14:50:15.702796 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:15.702798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:15.702828 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:50:15.702845 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:15.702846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:15.702854 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:15.702894 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:15.703006 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:15.703520 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:15.704169 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:15.704375 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:15.704426 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:15.704980 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:15.704981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:15.704982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:15.704986 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:15.704999 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:15.705003 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:15.705046 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:15.705086 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 14:50:15.705107 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:50:15.719746 21100 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:15.720490 21100 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:15.729879 21100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58d07d3691bd400991e24e4f322e8ffb: Generated new TSK 0
I20260620 14:50:15.730082 21100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:15.735653 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 21020
2026-06-20T14:50:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1224 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:15 dist-test-slave-cx89 krb5kdc[21119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:15 dist-test-slave-cx89 krb5kdc[21119](info): set up 2 sockets
Jun 20 14:50:15 dist-test-slave-cx89 krb5kdc[21119](info): commencing operation
krb5kdc: starting...
W20260620 14:50:16.691159 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:16 dist-test-slave-cx89 krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:50:16.801635 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33501
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:16.881117 21132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:16.881317 21132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:16.881340 21132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:16.882419 21132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:16.882458 21132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:16.882472 21132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:16.882481 21132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:16.882491 21132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:16.883932 21132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:16.884120 21132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:16.884280 21132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:16.888729 21138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:16.888808 21132 server_base.cc:1034] running on GCE node
W20260620 14:50:16.888759 21137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:16.888761 21140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:16.889117 21132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:16.889406 21132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:16.890550 21132 hybrid_clock.cc:648] HybridClock initialized: now 1781967016890535 us; error 31 us; skew 500 ppm
Jun 20 14:50:16 dist-test-slave-cx89 krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:16.892795 21132 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:50:16.893173 21132 webserver.cc:460] Webserver started at http://127.0.0.1:33353/ using document root <none> and password file <none>
I20260620 14:50:16.893357 21132 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:16.893402 21132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:16.893519 21132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:16.894502 21132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "39be1b2bb38c43e2ab143ae74a919097"
format_stamp: "Formatted at 2026-06-20 14:50:16 on dist-test-slave-cx89"
server_key: "1e24a18809b56e12fde68a1065699575"
server_key_iv: "a78db6cbf39f0e58eecc5f1f8517f4a5"
server_key_version: "encryptionkey@0"
I20260620 14:50:16.894819 21132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "39be1b2bb38c43e2ab143ae74a919097"
format_stamp: "Formatted at 2026-06-20 14:50:16 on dist-test-slave-cx89"
server_key: "1e24a18809b56e12fde68a1065699575"
server_key_iv: "a78db6cbf39f0e58eecc5f1f8517f4a5"
server_key_version: "encryptionkey@0"
I20260620 14:50:16.896036 21132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:50:16.896674 21147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:16.896807 21132 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:16.896886 21132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "39be1b2bb38c43e2ab143ae74a919097"
format_stamp: "Formatted at 2026-06-20 14:50:16 on dist-test-slave-cx89"
server_key: "1e24a18809b56e12fde68a1065699575"
server_key_iv: "a78db6cbf39f0e58eecc5f1f8517f4a5"
server_key_version: "encryptionkey@0"
I20260620 14:50:16.896944 21132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:16.900132 21132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:16.901835 21132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:16.901955 21132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:16.905485 21132 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33501
I20260620 14:50:16.905516 21199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33501 every 8 connection(s)
I20260620 14:50:16.905822 21132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:16.906697 21200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:16.908883 21200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097: Bootstrap starting.
I20260620 14:50:16.909406 21200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:16.909657 21200 log.cc:824] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:16.910254 21200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097: No bootstrap required, opened a new log
I20260620 14:50:16.911823 21200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39be1b2bb38c43e2ab143ae74a919097" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33501 } }
I20260620 14:50:16.911955 21200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:16.911975 21200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39be1b2bb38c43e2ab143ae74a919097, State: Initialized, Role: FOLLOWER
I20260620 14:50:16.912110 21200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39be1b2bb38c43e2ab143ae74a919097" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33501 } }
I20260620 14:50:16.912181 21200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:16.912214 21200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:16.912258 21200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:16.912743 21200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39be1b2bb38c43e2ab143ae74a919097" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33501 } }
I20260620 14:50:16.912830 21200 leader_election.cc:302] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39be1b2bb38c43e2ab143ae74a919097; no voters:
I20260620 14:50:16.912981 21200 leader_election.cc:288] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:16.913022 21203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:16.913178 21203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097 [term 1 LEADER]: Becoming Leader. State: Replica: 39be1b2bb38c43e2ab143ae74a919097, State: Running, Role: LEADER
I20260620 14:50:16.913208 21200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:16.913275 21203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39be1b2bb38c43e2ab143ae74a919097" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33501 } }
I20260620 14:50:16.913616 21205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39be1b2bb38c43e2ab143ae74a919097. Latest consensus state: current_term: 1 leader_uuid: "39be1b2bb38c43e2ab143ae74a919097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39be1b2bb38c43e2ab143ae74a919097" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33501 } } }
I20260620 14:50:16.913616 21204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39be1b2bb38c43e2ab143ae74a919097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39be1b2bb38c43e2ab143ae74a919097" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33501 } } }
I20260620 14:50:16.913712 21205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:16.913717 21204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:16.913887 21207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:16.914393 21207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:16.915828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 21132
I20260620 14:50:16.915951 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:16.915987 21207 catalog_manager.cc:1347] Generated new cluster ID: c5e0dbc52a114da3b05a6f20d8ad56b3
I20260620 14:50:16.916026 21207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:16.916097 392 external_mini_cluster.cc:1428] Setting key 340e8ba2239f4438d7cca03a4f43bf5f
I20260620 14:50:16.921092 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:16.916493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52485 (local address 127.0.0.1:33501)
0620 14:50:16.919693 (+ 3200us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:16.919701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:16.919725 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:50:16.919818 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:16.919824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:16.919857 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:16.919968 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:16.920662 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:16.920666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:16.920666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:16.920670 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:16.920706 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:16.920717 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:16.920768 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:16.920840 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 14:50:16.920961 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3158,"thread_start_us":62,"threads_started":1}
I20260620 14:50:16.922104 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:16.922467 21207 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:16.922608 21207 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:16.928458 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:16.923652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42655 (local address 127.0.0.1:33501)
0620 14:50:16.923685 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:16.923686 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:16.927604 (+ 3918us) server_negotiation.cc:407] Connection header received
0620 14:50:16.927668 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:16.927669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:16.927675 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:16.927724 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:16.928265 (+ 541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:16.928266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:16.928267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:16.928271 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:16.928284 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:16.928289 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:16.928334 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:16.928382 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:50:16.928394 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:50:16.931869 21207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39be1b2bb38c43e2ab143ae74a919097: Generated new TSK 0
I20260620 14:50:16.932034 21207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:16.935946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 21132
2026-06-20T14:50:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1200 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:16 dist-test-slave-cx89 krb5kdc[21231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:16 dist-test-slave-cx89 krb5kdc[21231](info): set up 2 sockets
Jun 20 14:50:16 dist-test-slave-cx89 krb5kdc[21231](info): commencing operation
krb5kdc: starting...
W20260620 14:50:17.883189 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:17 dist-test-slave-cx89 krb5kdc[21231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:50:17.994364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39719
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:18.072736 21244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:18.072942 21244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:18.072976 21244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:18.074107 21244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:18.074156 21244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:18.074177 21244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:18.074190 21244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:18.074206 21244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:18.075661 21244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:18.075894 21244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:18.076088 21244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:18.080514 21252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:18.080580 21249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:18.080542 21250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:18.080621 21244 server_base.cc:1034] running on GCE node
I20260620 14:50:18.080896 21244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:18.081153 21244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:18.082306 21244 hybrid_clock.cc:648] HybridClock initialized: now 1781967018082292 us; error 30 us; skew 500 ppm
Jun 20 14:50:18 dist-test-slave-cx89 krb5kdc[21231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:18.088395 21244 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:50:18.088826 21244 webserver.cc:460] Webserver started at http://127.0.0.1:35845/ using document root <none> and password file <none>
I20260620 14:50:18.089020 21244 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:18.089068 21244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:18.089192 21244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:18.090215 21244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "0119ce0cdd6d4f58a833699ec981a236"
format_stamp: "Formatted at 2026-06-20 14:50:18 on dist-test-slave-cx89"
server_key: "f51a2417d12bbac1ff3cd7313b49afb8"
server_key_iv: "2ff438f927b2c7bd19fa33244f4bd07e"
server_key_version: "encryptionkey@0"
I20260620 14:50:18.090538 21244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "0119ce0cdd6d4f58a833699ec981a236"
format_stamp: "Formatted at 2026-06-20 14:50:18 on dist-test-slave-cx89"
server_key: "f51a2417d12bbac1ff3cd7313b49afb8"
server_key_iv: "2ff438f927b2c7bd19fa33244f4bd07e"
server_key_version: "encryptionkey@0"
I20260620 14:50:18.091841 21244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:50:18.092528 21259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:18.092685 21244 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:18.092767 21244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "0119ce0cdd6d4f58a833699ec981a236"
format_stamp: "Formatted at 2026-06-20 14:50:18 on dist-test-slave-cx89"
server_key: "f51a2417d12bbac1ff3cd7313b49afb8"
server_key_iv: "2ff438f927b2c7bd19fa33244f4bd07e"
server_key_version: "encryptionkey@0"
I20260620 14:50:18.092826 21244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:18.150811 21244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:18.153018 21244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:18.153164 21244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:18.156736 21244 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39719
I20260620 14:50:18.156775 21311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39719 every 8 connection(s)
I20260620 14:50:18.157101 21244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:18.157898 21312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:18.159355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 21244
I20260620 14:50:18.159488 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:18.159667 392 external_mini_cluster.cc:1428] Setting key df300e3dfb0190ebd516fd1b11638592
I20260620 14:50:18.160468 21312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236: Bootstrap starting.
I20260620 14:50:18.161199 21312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:18.161532 21312 log.cc:824] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:18.162266 21312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236: No bootstrap required, opened a new log
I20260620 14:50:18.164383 21312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0119ce0cdd6d4f58a833699ec981a236" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39719 } }
I20260620 14:50:18.164563 21312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:18.164599 21312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0119ce0cdd6d4f58a833699ec981a236, State: Initialized, Role: FOLLOWER
I20260620 14:50:18.164734 21312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0119ce0cdd6d4f58a833699ec981a236" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39719 } }
I20260620 14:50:18.164801 21312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:18.164824 21312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:18.164866 21312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:18.165416 21312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0119ce0cdd6d4f58a833699ec981a236" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39719 } }
I20260620 14:50:18.165510 21312 leader_election.cc:302] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0119ce0cdd6d4f58a833699ec981a236; no voters:
I20260620 14:50:18.165655 21312 leader_election.cc:288] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:18.165691 21317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:18.165795 21317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236 [term 1 LEADER]: Becoming Leader. State: Replica: 0119ce0cdd6d4f58a833699ec981a236, State: Running, Role: LEADER
I20260620 14:50:18.165912 21312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:18.165937 21317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0119ce0cdd6d4f58a833699ec981a236" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39719 } }
I20260620 14:50:18.166276 21317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0119ce0cdd6d4f58a833699ec981a236. Latest consensus state: current_term: 1 leader_uuid: "0119ce0cdd6d4f58a833699ec981a236" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0119ce0cdd6d4f58a833699ec981a236" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39719 } } }
I20260620 14:50:18.166349 21317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236 [sys.catalog]: This master's current role is: LEADER
Jun 20 14:50:18 dist-test-slave-cx89 krb5kdc[21231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:50:18.166335 21318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0119ce0cdd6d4f58a833699ec981a236" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0119ce0cdd6d4f58a833699ec981a236" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39719 } } }
I20260620 14:50:18.166519 21318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:18.167191 21323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:18.167807 21323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:18.169157 21323 catalog_manager.cc:1347] Generated new cluster ID: 84993b81745844068f05f45d37a98947
I20260620 14:50:18.169205 21323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:18.170660 21315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:18.160228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:45865 (local address 127.0.0.1:39719)
0620 14:50:18.160393 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:18.160401 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:18.160428 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:50:18.160915 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:18.160922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:18.160976 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:18.161103 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:18.161903 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:18.162867 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:18.163970 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:18.164368 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:18.164444 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:18.166960 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:18.166964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:18.166964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:18.166976 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:18.168880 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:18.169313 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:18.169316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:18.169316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:18.169411 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:18.169665 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:18.169666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:18.169666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:18.169961 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:18.170114 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:18.170229 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 14:50:18.170439 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260620 14:50:18.171604 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:18.175498 21323 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:18.176137 21323 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:18.180047 21315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:18.173139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:42267 (local address 127.0.0.1:39719)
0620 14:50:18.173176 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:18.173177 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:18.173225 (+ 48us) server_negotiation.cc:407] Connection header received
0620 14:50:18.173297 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:18.173298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:18.173307 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:18.173348 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:18.173945 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:18.174728 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:18.175765 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:18.176069 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:18.176135 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:18.177427 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:18.177428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:18.177428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:18.177434 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:18.178876 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:18.179256 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:18.179257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:18.179258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:18.179306 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:18.179521 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:18.179522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:18.179522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:18.179630 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:18.179717 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:18.179875 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 14:50:18.179938 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":19}
I20260620 14:50:18.183099 21323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0119ce0cdd6d4f58a833699ec981a236: Generated new TSK 0
I20260620 14:50:18.183274 21323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:18.187841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 21244
W20260620 14:50:18.191630 21229 connection.cc:535] client connection to 127.0.0.1:39719 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39719): Connection reset by peer (error 104)
2026-06-20T14:50:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1252 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:18 dist-test-slave-cx89 krb5kdc[21343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:18 dist-test-slave-cx89 krb5kdc[21343](info): set up 2 sockets
Jun 20 14:50:18 dist-test-slave-cx89 krb5kdc[21343](info): commencing operation
krb5kdc: starting...
W20260620 14:50:19.139065 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:19 dist-test-slave-cx89 krb5kdc[21343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:50:19.246480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:19.324054 21356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:19.324252 21356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:19.324285 21356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:19.325403 21356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:19.325441 21356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:19.325460 21356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:19.325476 21356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:19.325492 21356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:19.326885 21356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:19.327113 21356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:19.327312 21356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:50:19.331899 21356 server_base.cc:1034] running on GCE node
W20260620 14:50:19.331867 21364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:19.331867 21361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:19.331858 21362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:19.332284 21356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:19.332548 21356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:19.333699 21356 hybrid_clock.cc:648] HybridClock initialized: now 1781967019333685 us; error 27 us; skew 500 ppm
Jun 20 14:50:19 dist-test-slave-cx89 krb5kdc[21343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:19.335974 21356 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:50:19.336345 21356 webserver.cc:460] Webserver started at http://127.0.0.1:35645/ using document root <none> and password file <none>
I20260620 14:50:19.336509 21356 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:19.336545 21356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:19.336630 21356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:19.337571 21356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "f8ea2373600c4fdd9b469f3d8e1a32b7"
format_stamp: "Formatted at 2026-06-20 14:50:19 on dist-test-slave-cx89"
server_key: "30971074030d12d2b46331ea9f86b4ed"
server_key_iv: "79bcdbf173574c99ead11a355c6124b2"
server_key_version: "encryptionkey@0"
I20260620 14:50:19.337854 21356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "f8ea2373600c4fdd9b469f3d8e1a32b7"
format_stamp: "Formatted at 2026-06-20 14:50:19 on dist-test-slave-cx89"
server_key: "30971074030d12d2b46331ea9f86b4ed"
server_key_iv: "79bcdbf173574c99ead11a355c6124b2"
server_key_version: "encryptionkey@0"
I20260620 14:50:19.338950 21356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:50:19.339596 21371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:19.339751 21356 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:19.339831 21356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "f8ea2373600c4fdd9b469f3d8e1a32b7"
format_stamp: "Formatted at 2026-06-20 14:50:19 on dist-test-slave-cx89"
server_key: "30971074030d12d2b46331ea9f86b4ed"
server_key_iv: "79bcdbf173574c99ead11a355c6124b2"
server_key_version: "encryptionkey@0"
I20260620 14:50:19.339895 21356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:19.348922 21356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:19.350440 21356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:19.350559 21356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:19.354127 21356 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44473
I20260620 14:50:19.354157 21423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44473 every 8 connection(s)
I20260620 14:50:19.354461 21356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:19.355225 21424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:19.357435 21424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7: Bootstrap starting.
I20260620 14:50:19.357986 21424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:19.358232 21424 log.cc:824] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:19.358914 21424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7: No bootstrap required, opened a new log
I20260620 14:50:19.360505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 21356
I20260620 14:50:19.360625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:19.360558 21424 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ea2373600c4fdd9b469f3d8e1a32b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44473 } }
I20260620 14:50:19.360723 21424 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:19.360764 21424 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ea2373600c4fdd9b469f3d8e1a32b7, State: Initialized, Role: FOLLOWER
I20260620 14:50:19.360805 392 external_mini_cluster.cc:1428] Setting key 1abd3a5e292738f89e491bc0b5ac9ec7
I20260620 14:50:19.360910 21424 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ea2373600c4fdd9b469f3d8e1a32b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44473 } }
I20260620 14:50:19.360987 21424 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:19.361008 21424 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:19.361054 21424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:19.362020 21424 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ea2373600c4fdd9b469f3d8e1a32b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44473 } }
I20260620 14:50:19.362112 21424 leader_election.cc:302] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8ea2373600c4fdd9b469f3d8e1a32b7; no voters:
I20260620 14:50:19.362268 21424 leader_election.cc:288] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:19.362306 21429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:19.362444 21429 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7 [term 1 LEADER]: Becoming Leader. State: Replica: f8ea2373600c4fdd9b469f3d8e1a32b7, State: Running, Role: LEADER
I20260620 14:50:19.362515 21424 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:19.362552 21429 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ea2373600c4fdd9b469f3d8e1a32b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44473 } }
I20260620 14:50:19.362876 21429 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8ea2373600c4fdd9b469f3d8e1a32b7. Latest consensus state: current_term: 1 leader_uuid: "f8ea2373600c4fdd9b469f3d8e1a32b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ea2373600c4fdd9b469f3d8e1a32b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44473 } } }
I20260620 14:50:19.363096 21429 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:19.363014 21430 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8ea2373600c4fdd9b469f3d8e1a32b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ea2373600c4fdd9b469f3d8e1a32b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44473 } } }
I20260620 14:50:19.363485 21430 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:19.363727 21438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:19.364223 21438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:19.366143 21438 catalog_manager.cc:1347] Generated new cluster ID: 479352b44fbb4874a4a511d96abb5044
I20260620 14:50:19.366199 21438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:19.366458 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:19.361239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:44999 (local address 127.0.0.1:44473)
0620 14:50:19.361403 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:19.361413 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:19.361445 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:50:19.361571 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:19.361579 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:19.361625 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:19.361731 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:19.361956 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:19.363233 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:19.364290 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:19.364885 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:19.364991 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:19.365654 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:19.365657 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:19.365658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:19.365858 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:19.365898 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:19.365913 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:19.365986 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:19.366230 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 14:50:19.366257 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260620 14:50:19.367460 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:19.374719 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:19.368870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:57157 (local address 127.0.0.1:44473)
0620 14:50:19.368909 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:19.368910 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:19.371567 (+ 2657us) server_negotiation.cc:407] Connection header received
0620 14:50:19.371598 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:19.371600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:19.371611 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:19.371677 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:19.371797 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:19.372507 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:19.373473 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:19.373793 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:19.373853 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:19.374482 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:19.374483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:19.374483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:19.374487 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:19.374503 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:19.374507 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:19.374573 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:19.374629 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 14:50:19.374642 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:50:19.386024 21438 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:19.386642 21438 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:19.394269 21438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8ea2373600c4fdd9b469f3d8e1a32b7: Generated new TSK 0
I20260620 14:50:19.394477 21438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:19.396518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 21356
W20260620 14:50:19.400298 21341 connection.cc:535] client connection to 127.0.0.1:44473 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44473): Connection reset by peer (error 104)
2026-06-20T14:50:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:19 dist-test-slave-cx89 krb5kdc[21455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:19 dist-test-slave-cx89 krb5kdc[21455](info): set up 2 sockets
Jun 20 14:50:19 dist-test-slave-cx89 krb5kdc[21455](info): commencing operation
krb5kdc: starting...
W20260620 14:50:20.343078 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:20 dist-test-slave-cx89 krb5kdc[21455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:50:20.451823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45971
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:20.528604 21468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:20.528800 21468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:20.528831 21468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:20.529920 21468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:20.529958 21468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:20.529978 21468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:20.529992 21468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:20.530019 21468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:20.531477 21468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:20.531706 21468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:20.531901 21468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:20.536229 21474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:20.536237 21476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:20.536244 21473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:20.536581 21468 server_base.cc:1034] running on GCE node
I20260620 14:50:20.536808 21468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:20.537058 21468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:20.538177 21468 hybrid_clock.cc:648] HybridClock initialized: now 1781967020538170 us; error 25 us; skew 500 ppm
Jun 20 14:50:20 dist-test-slave-cx89 krb5kdc[21455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:20.540447 21468 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:50:20.540851 21468 webserver.cc:460] Webserver started at http://127.0.0.1:37769/ using document root <none> and password file <none>
I20260620 14:50:20.541045 21468 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:20.541090 21468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:20.541217 21468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:20.542192 21468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "1c9d7283b1fc4aac8d4e1e1726c096b3"
format_stamp: "Formatted at 2026-06-20 14:50:20 on dist-test-slave-cx89"
server_key: "75d3616200430c0139cdf7b56fa7f4b6"
server_key_iv: "971ea059dfd1cf62d0163e413a3763a4"
server_key_version: "encryptionkey@0"
I20260620 14:50:20.542503 21468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "1c9d7283b1fc4aac8d4e1e1726c096b3"
format_stamp: "Formatted at 2026-06-20 14:50:20 on dist-test-slave-cx89"
server_key: "75d3616200430c0139cdf7b56fa7f4b6"
server_key_iv: "971ea059dfd1cf62d0163e413a3763a4"
server_key_version: "encryptionkey@0"
I20260620 14:50:20.543689 21468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:50:20.544299 21483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:20.544445 21468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:50:20.544524 21468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "1c9d7283b1fc4aac8d4e1e1726c096b3"
format_stamp: "Formatted at 2026-06-20 14:50:20 on dist-test-slave-cx89"
server_key: "75d3616200430c0139cdf7b56fa7f4b6"
server_key_iv: "971ea059dfd1cf62d0163e413a3763a4"
server_key_version: "encryptionkey@0"
I20260620 14:50:20.544580 21468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:20.547688 21468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:20.549365 21468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:20.549487 21468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:20.553002 21468 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45971
I20260620 14:50:20.553041 21535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45971 every 8 connection(s)
I20260620 14:50:20.553321 21468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:20.554180 21536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:20.555712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 21468
I20260620 14:50:20.555828 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:20.556007 392 external_mini_cluster.cc:1428] Setting key 5ff94b482a69262b13e7dd9f458dde9c
I20260620 14:50:20.556602 21536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3: Bootstrap starting.
I20260620 14:50:20.557130 21536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:20.557353 21536 log.cc:824] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:20.557946 21536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3: No bootstrap required, opened a new log
I20260620 14:50:20.559279 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:20.556471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:46989 (local address 127.0.0.1:45971)
0620 14:50:20.556626 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:20.556635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:20.557654 (+ 1019us) server_negotiation.cc:407] Connection header received
0620 14:50:20.558110 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:20.558118 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:20.558144 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:20.558277 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:20.558902 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:20.558905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:20.558906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:20.558910 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:20.558949 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:20.558959 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:20.559029 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:20.559077 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:50:20.559118 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260620 14:50:20.560184 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:20.560178 21536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c9d7283b1fc4aac8d4e1e1726c096b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45971 } }
I20260620 14:50:20.560338 21536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:20.560381 21536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c9d7283b1fc4aac8d4e1e1726c096b3, State: Initialized, Role: FOLLOWER
I20260620 14:50:20.560488 21536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c9d7283b1fc4aac8d4e1e1726c096b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45971 } }
I20260620 14:50:20.560571 21536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:20.560595 21536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:20.560640 21536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:20.561144 21536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c9d7283b1fc4aac8d4e1e1726c096b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45971 } }
I20260620 14:50:20.561246 21536 leader_election.cc:302] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c9d7283b1fc4aac8d4e1e1726c096b3; no voters:
I20260620 14:50:20.561399 21536 leader_election.cc:288] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:20.561432 21545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:20.561578 21545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3 [term 1 LEADER]: Becoming Leader. State: Replica: 1c9d7283b1fc4aac8d4e1e1726c096b3, State: Running, Role: LEADER
I20260620 14:50:20.561664 21536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:20.561697 21545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c9d7283b1fc4aac8d4e1e1726c096b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45971 } }
I20260620 14:50:20.562296 21545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c9d7283b1fc4aac8d4e1e1726c096b3. Latest consensus state: current_term: 1 leader_uuid: "1c9d7283b1fc4aac8d4e1e1726c096b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c9d7283b1fc4aac8d4e1e1726c096b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45971 } } }
I20260620 14:50:20.562376 21545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:20.562381 21547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c9d7283b1fc4aac8d4e1e1726c096b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c9d7283b1fc4aac8d4e1e1726c096b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45971 } } }
I20260620 14:50:20.562561 21547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:20.562697 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:20.561535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:35913 (local address 127.0.0.1:45971)
0620 14:50:20.561569 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:20.561571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:20.561662 (+ 91us) server_negotiation.cc:407] Connection header received
0620 14:50:20.561747 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:20.561747 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:20.561751 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:20.561803 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:20.562405 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:20.562406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:20.562407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:20.562410 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:20.562422 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:20.562426 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:20.562470 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:20.562635 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 14:50:20.562646 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:50:20.562798 21555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:20.563289 21555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:20.564739 21555 catalog_manager.cc:1347] Generated new cluster ID: 92ce75302396439d8ede3c811fc3fa78
I20260620 14:50:20.564781 21555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:20.580284 21555 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:20.580468 21555 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:20.583881 21555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c9d7283b1fc4aac8d4e1e1726c096b3: Generated new TSK 0
I20260620 14:50:20.584053 21555 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:20.590824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 21468
2026-06-20T14:50:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1193 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:20 dist-test-slave-cx89 krb5kdc[21567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:20 dist-test-slave-cx89 krb5kdc[21567](info): set up 2 sockets
Jun 20 14:50:20 dist-test-slave-cx89 krb5kdc[21567](info): commencing operation
krb5kdc: starting...
W20260620 14:50:21.531126 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:21 dist-test-slave-cx89 krb5kdc[21567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:50:21.639478 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41167
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:21.716818 21580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:21.717016 21580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:21.717038 21580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:21.718103 21580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:21.718137 21580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:21.718149 21580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:21.718160 21580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:21.718170 21580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:21.719594 21580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41167
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:21.719782 21580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:21.719937 21580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:21.724373 21585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:21.724433 21588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:21.724500 21580 server_base.cc:1034] running on GCE node
W20260620 14:50:21.724449 21586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:21.724814 21580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:21.725082 21580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:21.726244 21580 hybrid_clock.cc:648] HybridClock initialized: now 1781967021726228 us; error 31 us; skew 500 ppm
Jun 20 14:50:21 dist-test-slave-cx89 krb5kdc[21567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:21.728574 21580 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:50:21.728982 21580 webserver.cc:460] Webserver started at http://127.0.0.1:44305/ using document root <none> and password file <none>
I20260620 14:50:21.729192 21580 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:21.729238 21580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:21.729357 21580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:21.730345 21580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "b448dd5760364e3cb6e9d2cfbb68f182"
format_stamp: "Formatted at 2026-06-20 14:50:21 on dist-test-slave-cx89"
server_key: "91e60ef75d32ecd4e7f50724b5613582"
server_key_iv: "564d2951ac0572eddb0fdb184cdfe242"
server_key_version: "encryptionkey@0"
I20260620 14:50:21.730654 21580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "b448dd5760364e3cb6e9d2cfbb68f182"
format_stamp: "Formatted at 2026-06-20 14:50:21 on dist-test-slave-cx89"
server_key: "91e60ef75d32ecd4e7f50724b5613582"
server_key_iv: "564d2951ac0572eddb0fdb184cdfe242"
server_key_version: "encryptionkey@0"
I20260620 14:50:21.731897 21580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:50:21.732563 21595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:21.732707 21580 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:50:21.732784 21580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "b448dd5760364e3cb6e9d2cfbb68f182"
format_stamp: "Formatted at 2026-06-20 14:50:21 on dist-test-slave-cx89"
server_key: "91e60ef75d32ecd4e7f50724b5613582"
server_key_iv: "564d2951ac0572eddb0fdb184cdfe242"
server_key_version: "encryptionkey@0"
I20260620 14:50:21.732838 21580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:21.742220 21580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:21.744225 21580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:21.744349 21580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:21.747870 21580 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41167
I20260620 14:50:21.747903 21647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41167 every 8 connection(s)
I20260620 14:50:21.748206 21580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:21.748950 21648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:21.751019 21648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182: Bootstrap starting.
I20260620 14:50:21.751556 21648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:21.751783 21648 log.cc:824] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:21.752360 21648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182: No bootstrap required, opened a new log
I20260620 14:50:21.753369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 21580
I20260620 14:50:21.753484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:21.753650 392 external_mini_cluster.cc:1428] Setting key bbcc24dd7718c6fecddf2d0e9f4b1fa8
I20260620 14:50:21.754115 21648 raft_consensus.cc:348] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b448dd5760364e3cb6e9d2cfbb68f182" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41167 } }
I20260620 14:50:21.754302 21648 raft_consensus.cc:374] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:21.754331 21648 raft_consensus.cc:729] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b448dd5760364e3cb6e9d2cfbb68f182, State: Initialized, Role: FOLLOWER
I20260620 14:50:21.754458 21648 consensus_queue.cc:260] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b448dd5760364e3cb6e9d2cfbb68f182" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41167 } }
I20260620 14:50:21.754514 21648 raft_consensus.cc:388] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:21.754545 21648 raft_consensus.cc:482] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:21.754586 21648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:21.755609 21648 raft_consensus.cc:504] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b448dd5760364e3cb6e9d2cfbb68f182" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41167 } }
I20260620 14:50:21.755693 21648 leader_election.cc:302] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b448dd5760364e3cb6e9d2cfbb68f182; no voters:
I20260620 14:50:21.755836 21648 leader_election.cc:288] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:21.755856 21653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:21.755950 21653 raft_consensus.cc:686] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182 [term 1 LEADER]: Becoming Leader. State: Replica: b448dd5760364e3cb6e9d2cfbb68f182, State: Running, Role: LEADER
I20260620 14:50:21.756073 21648 sys_catalog.cc:564] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:21.756079 21653 consensus_queue.cc:237] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b448dd5760364e3cb6e9d2cfbb68f182" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41167 } }
I20260620 14:50:21.756418 21653 sys_catalog.cc:455] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b448dd5760364e3cb6e9d2cfbb68f182. Latest consensus state: current_term: 1 leader_uuid: "b448dd5760364e3cb6e9d2cfbb68f182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b448dd5760364e3cb6e9d2cfbb68f182" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41167 } } }
I20260620 14:50:21.756428 21654 sys_catalog.cc:455] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b448dd5760364e3cb6e9d2cfbb68f182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b448dd5760364e3cb6e9d2cfbb68f182" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41167 } } }
I20260620 14:50:21.756564 21653 sys_catalog.cc:458] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:21.756575 21654 sys_catalog.cc:458] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:21.757350 21660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:21.757921 21660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:21.759438 21660 catalog_manager.cc:1347] Generated new cluster ID: ed8a5c6c5d28415aa67442e329e94088
I20260620 14:50:21.759481 21660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:50:21 dist-test-slave-cx89 krb5kdc[21567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:50:21.764341 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:21.754135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:53607 (local address 127.0.0.1:41167)
0620 14:50:21.754293 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:21.754302 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:21.754334 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:50:21.754449 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:21.754456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:21.754511 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:21.754661 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:21.755363 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:21.756582 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:21.757596 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:21.758003 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:21.758109 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:21.760683 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:21.760687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:21.760687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:21.760698 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:21.762478 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:21.762897 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:21.762899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:21.762900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:21.762999 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:21.763210 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:21.763211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:21.763211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:21.763508 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:21.763653 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:21.764064 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 14:50:21.764156 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260620 14:50:21.765470 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:21.770524 21660 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:21.770968 21660 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:21.773640 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:21.766820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:56935 (local address 127.0.0.1:41167)
0620 14:50:21.766851 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:21.766853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:21.767034 (+ 181us) server_negotiation.cc:407] Connection header received
0620 14:50:21.767091 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:21.767091 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:21.767100 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:21.767144 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:21.767744 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:21.768429 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:21.769410 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:21.769718 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:21.769779 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:21.771159 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:21.771160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:21.771160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:21.771166 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:21.772576 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:21.772958 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:21.772959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:21.772959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:21.773003 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:21.773191 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:21.773191 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:21.773191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:21.773293 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:21.773384 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:21.773501 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 14:50:21.773560 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:50:21.774930 21660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b448dd5760364e3cb6e9d2cfbb68f182: Generated new TSK 0
I20260620 14:50:21.775086 21660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:21.777083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 21580
W20260620 14:50:21.780812 21565 connection.cc:535] client connection to 127.0.0.1:41167 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41167): Connection reset by peer (error 104)
2026-06-20T14:50:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1186 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:21 dist-test-slave-cx89 krb5kdc[21679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:21 dist-test-slave-cx89 krb5kdc[21679](info): set up 2 sockets
Jun 20 14:50:21 dist-test-slave-cx89 krb5kdc[21679](info): commencing operation
krb5kdc: starting...
W20260620 14:50:22.723170 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:22 dist-test-slave-cx89 krb5kdc[21679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:50:22.834326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37511
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:22.912635 21692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:22.912825 21692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:22.912845 21692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:22.913944 21692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:22.913983 21692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:22.913997 21692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:22.914019 21692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:22.914042 21692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:22.915501 21692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37511
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:22.915704 21692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:22.915905 21692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:22.920348 21697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:22.920403 21698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:22.920364 21700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:22.920397 21692 server_base.cc:1034] running on GCE node
I20260620 14:50:22.920742 21692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:22.921005 21692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:22.922142 21692 hybrid_clock.cc:648] HybridClock initialized: now 1781967022922123 us; error 27 us; skew 500 ppm
Jun 20 14:50:22 dist-test-slave-cx89 krb5kdc[21679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967022, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:22.924391 21692 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:50:22.924803 21692 webserver.cc:460] Webserver started at http://127.0.0.1:41609/ using document root <none> and password file <none>
I20260620 14:50:22.925009 21692 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:22.925055 21692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:22.925181 21692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:22.926152 21692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "af72ecc8cecf4516add1da18b7e64137"
format_stamp: "Formatted at 2026-06-20 14:50:22 on dist-test-slave-cx89"
server_key: "e06b88d6822571e48e94e1dc4ebb5966"
server_key_iv: "2c74f66ae8dd316d5d0f67662628db88"
server_key_version: "encryptionkey@0"
I20260620 14:50:22.926476 21692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "af72ecc8cecf4516add1da18b7e64137"
format_stamp: "Formatted at 2026-06-20 14:50:22 on dist-test-slave-cx89"
server_key: "e06b88d6822571e48e94e1dc4ebb5966"
server_key_iv: "2c74f66ae8dd316d5d0f67662628db88"
server_key_version: "encryptionkey@0"
I20260620 14:50:22.927670 21692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:50:22.928321 21707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:22.928447 21692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:22.928524 21692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "af72ecc8cecf4516add1da18b7e64137"
format_stamp: "Formatted at 2026-06-20 14:50:22 on dist-test-slave-cx89"
server_key: "e06b88d6822571e48e94e1dc4ebb5966"
server_key_iv: "2c74f66ae8dd316d5d0f67662628db88"
server_key_version: "encryptionkey@0"
I20260620 14:50:22.928582 21692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:22.937674 21692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:22.939666 21692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:22.939790 21692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:22.943452 21692 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37511
I20260620 14:50:22.943532 21759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37511 every 8 connection(s)
I20260620 14:50:22.943810 21692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:22.944674 21760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:22.946803 21760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137: Bootstrap starting.
I20260620 14:50:22.947327 21760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:22.947589 21760 log.cc:824] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:22.948218 21760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137: No bootstrap required, opened a new log
I20260620 14:50:22.948442 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 21692
I20260620 14:50:22.948542 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:22.948704 392 external_mini_cluster.cc:1428] Setting key ca41a2fca80f5bcea4becbf66491734c
I20260620 14:50:22.950276 21760 raft_consensus.cc:348] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af72ecc8cecf4516add1da18b7e64137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37511 } }
I20260620 14:50:22.950443 21760 raft_consensus.cc:374] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:22.950487 21760 raft_consensus.cc:729] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af72ecc8cecf4516add1da18b7e64137, State: Initialized, Role: FOLLOWER
I20260620 14:50:22.950615 21760 consensus_queue.cc:260] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af72ecc8cecf4516add1da18b7e64137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37511 } }
I20260620 14:50:22.950690 21760 raft_consensus.cc:388] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:22.950717 21760 raft_consensus.cc:482] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:22.950752 21760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:22.951400 21760 raft_consensus.cc:504] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af72ecc8cecf4516add1da18b7e64137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37511 } }
I20260620 14:50:22.951522 21760 leader_election.cc:302] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af72ecc8cecf4516add1da18b7e64137; no voters:
I20260620 14:50:22.951665 21760 leader_election.cc:288] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:22.951711 21765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:22.951881 21760 sys_catalog.cc:564] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:22.951898 21765 raft_consensus.cc:686] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137 [term 1 LEADER]: Becoming Leader. State: Replica: af72ecc8cecf4516add1da18b7e64137, State: Running, Role: LEADER
I20260620 14:50:22.952049 21765 consensus_queue.cc:237] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af72ecc8cecf4516add1da18b7e64137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37511 } }
I20260620 14:50:22.952394 21766 sys_catalog.cc:455] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af72ecc8cecf4516add1da18b7e64137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af72ecc8cecf4516add1da18b7e64137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37511 } } }
I20260620 14:50:22.952464 21766 sys_catalog.cc:458] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:22.952656 21767 sys_catalog.cc:455] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af72ecc8cecf4516add1da18b7e64137. Latest consensus state: current_term: 1 leader_uuid: "af72ecc8cecf4516add1da18b7e64137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af72ecc8cecf4516add1da18b7e64137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37511 } } }
I20260620 14:50:22.952910 21767 sys_catalog.cc:458] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:22.953074 21774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:22.953557 21774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:22.955390 21764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:22.949195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:56049 (local address 127.0.0.1:37511)
0620 14:50:22.949380 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:22.949390 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:22.949420 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:50:22.949865 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:22.949873 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:22.949935 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:22.950079 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:22.950274 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:22.951872 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:22.952985 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:22.953640 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:22.953738 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:22.954775 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:22.954778 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:22.954779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:22.954969 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:22.955006 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:22.955020 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:22.955101 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:22.955197 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 14:50:22.955226 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
I20260620 14:50:22.955706 21774 catalog_manager.cc:1347] Generated new cluster ID: 13288dc9edc24cebbdbc198a81454eee
I20260620 14:50:22.955742 21774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:22.956285 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:22.962230 21774 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:22.962905 21774 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:22.963001 21764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:22.959685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:57497 (local address 127.0.0.1:37511)
0620 14:50:22.959726 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:22.959728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:22.959832 (+ 104us) server_negotiation.cc:407] Connection header received
0620 14:50:22.959900 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:22.959901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:22.959910 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:22.959963 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:22.960100 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:22.960810 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:22.961786 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:22.962003 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:22.962062 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:22.962735 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:22.962736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:22.962737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:22.962741 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:22.962755 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:22.962759 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:22.962809 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:22.962871 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 14:50:22.962879 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:50:22.968057 21774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af72ecc8cecf4516add1da18b7e64137: Generated new TSK 0
I20260620 14:50:22.968232 21774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:22.973062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 21692
W20260620 14:50:22.976675 21677 connection.cc:535] client connection to 127.0.0.1:37511 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37511): Connection reset by peer (error 104)
2026-06-20T14:50:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1195 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:23 dist-test-slave-cx89 krb5kdc[21791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:23 dist-test-slave-cx89 krb5kdc[21791](info): set up 2 sockets
Jun 20 14:50:23 dist-test-slave-cx89 krb5kdc[21791](info): commencing operation
krb5kdc: starting...
W20260620 14:50:23.923346 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:23 dist-test-slave-cx89 krb5kdc[21791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:50:24.034232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:24.112006 21804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:24.112208 21804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:24.112229 21804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:24.113320 21804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:24.113355 21804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:24.113368 21804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:24.113379 21804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:24.113389 21804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:24.114805 21804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43955
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:24.115003 21804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:24.115172 21804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:24.119689 21812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:24.119724 21809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:24.119699 21810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:24.119730 21804 server_base.cc:1034] running on GCE node
I20260620 14:50:24.120128 21804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:24.120391 21804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:24.121553 21804 hybrid_clock.cc:648] HybridClock initialized: now 1781967024121541 us; error 26 us; skew 500 ppm
Jun 20 14:50:24 dist-test-slave-cx89 krb5kdc[21791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:24.127116 21804 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:50:24.127595 21804 webserver.cc:460] Webserver started at http://127.0.0.1:38403/ using document root <none> and password file <none>
I20260620 14:50:24.127815 21804 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:24.127862 21804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:24.127985 21804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:24.129015 21804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "70a726a2fae041d5a0f0f6516c036658"
format_stamp: "Formatted at 2026-06-20 14:50:24 on dist-test-slave-cx89"
server_key: "1f66850cd16fbaf51e21767cbd7ab6ae"
server_key_iv: "7cfdd3fbd802bc7854483028dde80936"
server_key_version: "encryptionkey@0"
I20260620 14:50:24.129346 21804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "70a726a2fae041d5a0f0f6516c036658"
format_stamp: "Formatted at 2026-06-20 14:50:24 on dist-test-slave-cx89"
server_key: "1f66850cd16fbaf51e21767cbd7ab6ae"
server_key_iv: "7cfdd3fbd802bc7854483028dde80936"
server_key_version: "encryptionkey@0"
I20260620 14:50:24.130560 21804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:50:24.131242 21819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:24.131405 21804 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:24.131517 21804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "70a726a2fae041d5a0f0f6516c036658"
format_stamp: "Formatted at 2026-06-20 14:50:24 on dist-test-slave-cx89"
server_key: "1f66850cd16fbaf51e21767cbd7ab6ae"
server_key_iv: "7cfdd3fbd802bc7854483028dde80936"
server_key_version: "encryptionkey@0"
I20260620 14:50:24.131582 21804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:24.134688 21804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:24.136608 21804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:24.136739 21804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:24.140511 21804 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43955
I20260620 14:50:24.140547 21871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43955 every 8 connection(s)
I20260620 14:50:24.140834 21804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:24.141703 21872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:24.143857 21872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658: Bootstrap starting.
I20260620 14:50:24.144357 21872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:24.144579 21872 log.cc:824] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:24.145159 21872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658: No bootstrap required, opened a new log
I20260620 14:50:24.146948 21872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a726a2fae041d5a0f0f6516c036658" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43955 } }
I20260620 14:50:24.147089 21872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:24.147110 21872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70a726a2fae041d5a0f0f6516c036658, State: Initialized, Role: FOLLOWER
I20260620 14:50:24.147218 21872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a726a2fae041d5a0f0f6516c036658" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43955 } }
I20260620 14:50:24.147266 21872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:24.147289 21872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:24.147321 21872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:24.147846 21872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a726a2fae041d5a0f0f6516c036658" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43955 } }
I20260620 14:50:24.147939 21872 leader_election.cc:302] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70a726a2fae041d5a0f0f6516c036658; no voters:
I20260620 14:50:24.148056 21872 leader_election.cc:288] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:24.148109 21875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:24.148269 21872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:24.148279 21875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658 [term 1 LEADER]: Becoming Leader. State: Replica: 70a726a2fae041d5a0f0f6516c036658, State: Running, Role: LEADER
I20260620 14:50:24.148468 21875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a726a2fae041d5a0f0f6516c036658" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43955 } }
I20260620 14:50:24.148522 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 21804
I20260620 14:50:24.148628 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:24.148803 392 external_mini_cluster.cc:1428] Setting key 354caf26fb4590df340b5c5697509c84
I20260620 14:50:24.149080 21877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70a726a2fae041d5a0f0f6516c036658. Latest consensus state: current_term: 1 leader_uuid: "70a726a2fae041d5a0f0f6516c036658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a726a2fae041d5a0f0f6516c036658" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43955 } } }
I20260620 14:50:24.149159 21877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:24.149240 21876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70a726a2fae041d5a0f0f6516c036658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a726a2fae041d5a0f0f6516c036658" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43955 } } }
I20260620 14:50:24.149322 21876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658 [sys.catalog]: This master's current role is: LEADER
W20260620 14:50:24.150084 21892 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:50:24.150177 21892 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:50:24.151140 21884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:24.151315 21884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:24.151362 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:24.149440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:43831 (local address 127.0.0.1:43955)
0620 14:50:24.149568 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:24.149579 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:24.149616 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:50:24.149792 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:24.149800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:24.149831 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:24.149946 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:24.150930 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:24.150934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:24.150934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:24.150939 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:24.150981 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:24.150996 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:24.151036 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:24.151142 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 14:50:24.151195 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":57,"threads_started":1}
I20260620 14:50:24.152163 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:24.152927 21884 catalog_manager.cc:1347] Generated new cluster ID: b93aea7291e749a9bb25a47d60c79a0e
I20260620 14:50:24.152976 21884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:24.156780 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:24.155721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:46823 (local address 127.0.0.1:43955)
0620 14:50:24.155755 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:24.155756 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:24.155868 (+ 112us) server_negotiation.cc:407] Connection header received
0620 14:50:24.155949 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:24.155949 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:24.155954 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:24.155977 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:24.156566 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:24.156567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:24.156567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:24.156570 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:24.156582 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:24.156586 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:24.156610 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:24.156696 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 14:50:24.156709 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 14:50:24.183600 21884 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:24.183801 21884 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:24.188834 21884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70a726a2fae041d5a0f0f6516c036658: Generated new TSK 0
I20260620 14:50:24.189029 21884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:24.190670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 21804
2026-06-20T14:50:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1217 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:24 dist-test-slave-cx89 krb5kdc[21903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:24 dist-test-slave-cx89 krb5kdc[21903](info): set up 2 sockets
Jun 20 14:50:24 dist-test-slave-cx89 krb5kdc[21903](info): commencing operation
krb5kdc: starting...
W20260620 14:50:25.151183 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:25 dist-test-slave-cx89 krb5kdc[21903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:50:25.260249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33713
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:25.338176 21916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:25.338402 21916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:25.338433 21916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:25.339586 21916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:25.339627 21916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:25.339644 21916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:25.339660 21916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:25.339676 21916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:25.341117 21916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33713
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:25.341357 21916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:25.341555 21916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:25.346016 21922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:25.346155 21916 server_base.cc:1034] running on GCE node
W20260620 14:50:25.346028 21924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:25.346196 21921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:25.346504 21916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:25.346776 21916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:25.347920 21916 hybrid_clock.cc:648] HybridClock initialized: now 1781967025347903 us; error 27 us; skew 500 ppm
Jun 20 14:50:25 dist-test-slave-cx89 krb5kdc[21903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:25.350225 21916 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:50:25.350629 21916 webserver.cc:460] Webserver started at http://127.0.98.62:38441/ using document root <none> and password file <none>
I20260620 14:50:25.350821 21916 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:25.350862 21916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:25.350970 21916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:25.352007 21916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "80e942f44ef845b697008769f782aeff"
format_stamp: "Formatted at 2026-06-20 14:50:25 on dist-test-slave-cx89"
server_key: "9f31f964f2259c8befa1a7076259deb2"
server_key_iv: "6295424037f36e7fb5baaa827c388b41"
server_key_version: "encryptionkey@0"
I20260620 14:50:25.352329 21916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "80e942f44ef845b697008769f782aeff"
format_stamp: "Formatted at 2026-06-20 14:50:25 on dist-test-slave-cx89"
server_key: "9f31f964f2259c8befa1a7076259deb2"
server_key_iv: "6295424037f36e7fb5baaa827c388b41"
server_key_version: "encryptionkey@0"
I20260620 14:50:25.353513 21916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:50:25.354157 21931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:25.354312 21916 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:25.354388 21916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "80e942f44ef845b697008769f782aeff"
format_stamp: "Formatted at 2026-06-20 14:50:25 on dist-test-slave-cx89"
server_key: "9f31f964f2259c8befa1a7076259deb2"
server_key_iv: "6295424037f36e7fb5baaa827c388b41"
server_key_version: "encryptionkey@0"
I20260620 14:50:25.354444 21916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:25.369159 21916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:25.371205 21916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:25.371335 21916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:25.374991 21916 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33713
I20260620 14:50:25.375036 21983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33713 every 8 connection(s)
I20260620 14:50:25.375360 21916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:25.376142 21984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:25.378316 21984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff: Bootstrap starting.
I20260620 14:50:25.378813 21984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:25.379029 21984 log.cc:824] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:25.379671 21984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff: No bootstrap required, opened a new log
I20260620 14:50:25.381171 21984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80e942f44ef845b697008769f782aeff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33713 } }
I20260620 14:50:25.381319 21984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:25.381340 21984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80e942f44ef845b697008769f782aeff, State: Initialized, Role: FOLLOWER
I20260620 14:50:25.381448 21984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80e942f44ef845b697008769f782aeff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33713 } }
I20260620 14:50:25.381517 21984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:25.381543 21984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:25.381575 21984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:25.382040 21984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80e942f44ef845b697008769f782aeff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33713 } }
I20260620 14:50:25.382120 21984 leader_election.cc:302] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80e942f44ef845b697008769f782aeff; no voters:
I20260620 14:50:25.382246 21984 leader_election.cc:288] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:25.382296 21987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:25.382462 21984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:25.382470 21987 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff [term 1 LEADER]: Becoming Leader. State: Replica: 80e942f44ef845b697008769f782aeff, State: Running, Role: LEADER
I20260620 14:50:25.382622 21987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80e942f44ef845b697008769f782aeff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33713 } }
I20260620 14:50:25.383111 21989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80e942f44ef845b697008769f782aeff. Latest consensus state: current_term: 1 leader_uuid: "80e942f44ef845b697008769f782aeff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80e942f44ef845b697008769f782aeff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33713 } } }
I20260620 14:50:25.383196 21989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:25.383476 21988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80e942f44ef845b697008769f782aeff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80e942f44ef845b697008769f782aeff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33713 } } }
I20260620 14:50:25.383579 21988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:25.383775 22000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:25.384254 22000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:25.384811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 21916
I20260620 14:50:25.384903 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:25.385061 392 external_mini_cluster.cc:1428] Setting key b51bd34ed80fb6a1c58b8d2d4873f498
I20260620 14:50:25.385891 22000 catalog_manager.cc:1347] Generated new cluster ID: ec220c33cd7b41e8be90495358724277
I20260620 14:50:25.385941 22000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:50:25 dist-test-slave-cx89 krb5kdc[21903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:50:25.396466 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:25.385638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46659 (local address 127.0.98.62:33713)
0620 14:50:25.385792 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:25.385802 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:25.385833 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:50:25.385950 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:25.385958 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:25.386013 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:25.386146 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:25.386918 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:25.388326 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:25.389415 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:25.390008 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:25.390096 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:25.392927 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:25.392931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:25.392932 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:25.392944 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:25.394825 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:25.395240 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:25.395243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:25.395243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:25.395364 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:25.395572 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:25.395573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:25.395573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:25.395895 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:25.396044 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:25.396124 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:50:25.396234 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260620 14:50:25.397426 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:25.405601 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:25.398819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54129 (local address 127.0.98.62:33713)
0620 14:50:25.398866 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:25.398868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:25.399013 (+ 145us) server_negotiation.cc:407] Connection header received
0620 14:50:25.399057 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:25.399058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:25.399067 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:25.399128 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:25.399716 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:25.400478 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:25.401414 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:25.401741 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:25.401800 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:25.403041 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:25.403043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:25.403043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:25.403050 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:25.404515 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:25.404880 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:25.404881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:25.404882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:25.404938 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:25.405120 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:25.405121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:25.405121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:25.405251 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:25.405357 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:25.405447 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 14:50:25.405506 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:50:25.421455 22000 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:25.421986 22000 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:25.428601 22000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80e942f44ef845b697008769f782aeff: Generated new TSK 0
I20260620 14:50:25.428793 22000 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:25.431090 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 21916
W20260620 14:50:25.434954 21901 connection.cc:535] client connection to 127.0.98.62:33713 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:33713): Connection reset by peer (error 104)
2026-06-20T14:50:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1240 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:25 dist-test-slave-cx89 krb5kdc[22015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:25 dist-test-slave-cx89 krb5kdc[22015](info): set up 2 sockets
Jun 20 14:50:25 dist-test-slave-cx89 krb5kdc[22015](info): commencing operation
krb5kdc: starting...
W20260620 14:50:26.375154 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:26 dist-test-slave-cx89 krb5kdc[22015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:50:26.483968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46849
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:26.561272 22028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:26.561479 22028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:26.561510 22028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:26.562646 22028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:26.562687 22028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:26.562705 22028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:26.562721 22028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:26.562747 22028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:26.564232 22028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46849
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:26.564467 22028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:26.564663 22028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:50:26.569152 22028 server_base.cc:1034] running on GCE node
W20260620 14:50:26.569119 22034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:26.569133 22033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:26.569133 22036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:26.569571 22028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:26.569847 22028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:26.570999 22028 hybrid_clock.cc:648] HybridClock initialized: now 1781967026570987 us; error 33 us; skew 500 ppm
Jun 20 14:50:26 dist-test-slave-cx89 krb5kdc[22015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:26.573328 22028 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:50:26.573746 22028 webserver.cc:460] Webserver started at http://127.0.98.62:45803/ using document root <none> and password file <none>
I20260620 14:50:26.573938 22028 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:26.573983 22028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:26.574101 22028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:26.575133 22028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "71592923a2d742f78c4d55e97b2d4f58"
format_stamp: "Formatted at 2026-06-20 14:50:26 on dist-test-slave-cx89"
server_key: "eb5ebde03378f4814590044687493c16"
server_key_iv: "134bea9c817af6f3e38ae0e19ae8cf48"
server_key_version: "encryptionkey@0"
I20260620 14:50:26.575502 22028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "71592923a2d742f78c4d55e97b2d4f58"
format_stamp: "Formatted at 2026-06-20 14:50:26 on dist-test-slave-cx89"
server_key: "eb5ebde03378f4814590044687493c16"
server_key_iv: "134bea9c817af6f3e38ae0e19ae8cf48"
server_key_version: "encryptionkey@0"
I20260620 14:50:26.576797 22028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:50:26.577577 22043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:26.577761 22028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:50:26.577852 22028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "71592923a2d742f78c4d55e97b2d4f58"
format_stamp: "Formatted at 2026-06-20 14:50:26 on dist-test-slave-cx89"
server_key: "eb5ebde03378f4814590044687493c16"
server_key_iv: "134bea9c817af6f3e38ae0e19ae8cf48"
server_key_version: "encryptionkey@0"
I20260620 14:50:26.577925 22028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:26.588881 22028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:26.590628 22028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:26.590762 22028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:26.594307 22028 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46849
I20260620 14:50:26.594347 22095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46849 every 8 connection(s)
I20260620 14:50:26.594648 22028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:26.595505 22096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:26.597709 22096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58: Bootstrap starting.
I20260620 14:50:26.598076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 22028
I20260620 14:50:26.598172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:26.598291 22096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:26.598347 392 external_mini_cluster.cc:1428] Setting key c17497ca1952deab6fba2e6cad63163c
I20260620 14:50:26.598552 22096 log.cc:824] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:26.599215 22096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58: No bootstrap required, opened a new log
I20260620 14:50:26.601554 22096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71592923a2d742f78c4d55e97b2d4f58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46849 } }
I20260620 14:50:26.601723 22096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:26.601758 22096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71592923a2d742f78c4d55e97b2d4f58, State: Initialized, Role: FOLLOWER
I20260620 14:50:26.601902 22096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71592923a2d742f78c4d55e97b2d4f58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46849 } }
I20260620 14:50:26.601980 22096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:26.602013 22096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:26.602054 22096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:26.602715 22096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71592923a2d742f78c4d55e97b2d4f58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46849 } }
I20260620 14:50:26.602813 22096 leader_election.cc:302] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71592923a2d742f78c4d55e97b2d4f58; no voters:
I20260620 14:50:26.602962 22096 leader_election.cc:288] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:26.602998 22101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:26.603132 22101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58 [term 1 LEADER]: Becoming Leader. State: Replica: 71592923a2d742f78c4d55e97b2d4f58, State: Running, Role: LEADER
I20260620 14:50:26.603207 22096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:26.603258 22101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71592923a2d742f78c4d55e97b2d4f58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46849 } }
I20260620 14:50:26.603650 22101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71592923a2d742f78c4d55e97b2d4f58. Latest consensus state: current_term: 1 leader_uuid: "71592923a2d742f78c4d55e97b2d4f58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71592923a2d742f78c4d55e97b2d4f58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46849 } } }
I20260620 14:50:26.603895 22101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:26.603986 22100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:26.598801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32867 (local address 127.0.98.62:46849)
0620 14:50:26.598958 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:26.598968 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:26.598996 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:50:26.599466 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:26.599474 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:26.599523 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:26.599642 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:26.599873 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:26.600738 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:26.601840 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:26.602433 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:26.602548 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:26.603235 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:26.603239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:26.603240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:26.603468 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:26.603507 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:26.603521 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:26.603569 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:26.603723 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 14:50:26.603750 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260620 14:50:26.603817 22103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71592923a2d742f78c4d55e97b2d4f58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71592923a2d742f78c4d55e97b2d4f58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46849 } } }
I20260620 14:50:26.604204 22103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:26.604405 22110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:26.604904 22110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:26.605106 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:26.606753 22110 catalog_manager.cc:1347] Generated new cluster ID: d23b6e486297419cbb2e56c4bc6962b1
I20260620 14:50:26.606804 22110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:26.609148 22100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:26.606484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41325 (local address 127.0.98.62:46849)
0620 14:50:26.606520 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:26.606522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:26.606612 (+ 90us) server_negotiation.cc:407] Connection header received
0620 14:50:26.606687 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:26.606688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:26.606697 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:26.606722 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:26.606906 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:26.607383 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:26.608079 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:26.608306 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:26.608360 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:26.608938 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:26.608940 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:26.608940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:26.608944 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:26.608957 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:26.608960 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:26.609005 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:26.609069 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 14:50:26.609077 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:50:26.640945 22110 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:26.641489 22110 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:26.650133 22110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71592923a2d742f78c4d55e97b2d4f58: Generated new TSK 0
I20260620 14:50:26.650323 22110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:26.652400 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 22028
W20260620 14:50:26.656251 22013 connection.cc:535] client connection to 127.0.98.62:46849 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:46849): Connection reset by peer (error 104)
2026-06-20T14:50:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:26 dist-test-slave-cx89 krb5kdc[22127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:26 dist-test-slave-cx89 krb5kdc[22127](info): set up 2 sockets
Jun 20 14:50:26 dist-test-slave-cx89 krb5kdc[22127](info): commencing operation
krb5kdc: starting...
W20260620 14:50:27.599222 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:27 dist-test-slave-cx89 krb5kdc[22127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:50:27.706241 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39371
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:27.790731 22140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:27.790938 22140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:27.790959 22140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:27.792186 22140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:27.792232 22140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:27.792244 22140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:27.792256 22140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:27.792266 22140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:27.793718 22140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39371
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:27.793939 22140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:27.794138 22140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:27.798607 22145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:27.798651 22140 server_base.cc:1034] running on GCE node
W20260620 14:50:27.798619 22148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:27.798622 22146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:27.798985 22140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:27.799257 22140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:27.800391 22140 hybrid_clock.cc:648] HybridClock initialized: now 1781967027800368 us; error 36 us; skew 500 ppm
Jun 20 14:50:27 dist-test-slave-cx89 krb5kdc[22127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:27.802606 22140 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:50:27.803053 22140 webserver.cc:460] Webserver started at http://127.0.98.62:40743/ using document root <none> and password file <none>
I20260620 14:50:27.803260 22140 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:27.803306 22140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:27.803452 22140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:27.804445 22140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "72977e0a06114142b97bc12d107b2ef8"
format_stamp: "Formatted at 2026-06-20 14:50:27 on dist-test-slave-cx89"
server_key: "1ef1d228651381ab6580d77bdff6696e"
server_key_iv: "2452a0cab52eaf83d42892e29f50ffe3"
server_key_version: "encryptionkey@0"
I20260620 14:50:27.804756 22140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "72977e0a06114142b97bc12d107b2ef8"
format_stamp: "Formatted at 2026-06-20 14:50:27 on dist-test-slave-cx89"
server_key: "1ef1d228651381ab6580d77bdff6696e"
server_key_iv: "2452a0cab52eaf83d42892e29f50ffe3"
server_key_version: "encryptionkey@0"
I20260620 14:50:27.805948 22140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:50:27.806618 22155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:27.806768 22140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:50:27.806847 22140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "72977e0a06114142b97bc12d107b2ef8"
format_stamp: "Formatted at 2026-06-20 14:50:27 on dist-test-slave-cx89"
server_key: "1ef1d228651381ab6580d77bdff6696e"
server_key_iv: "2452a0cab52eaf83d42892e29f50ffe3"
server_key_version: "encryptionkey@0"
I20260620 14:50:27.806919 22140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:27.810102 22140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:27.812107 22140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:27.812234 22140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:27.815868 22140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39371
I20260620 14:50:27.815915 22207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39371 every 8 connection(s)
I20260620 14:50:27.816226 22140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:27.817101 22208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:27.819295 22208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8: Bootstrap starting.
I20260620 14:50:27.819852 22208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:27.820137 22208 log.cc:824] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:27.820233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 22140
I20260620 14:50:27.820350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:27.820514 392 external_mini_cluster.cc:1428] Setting key 34dbf8024f39ab814faafd51f5dc4344
I20260620 14:50:27.820896 22208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8: No bootstrap required, opened a new log
I20260620 14:50:27.822870 22208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72977e0a06114142b97bc12d107b2ef8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39371 } }
I20260620 14:50:27.823045 22208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:27.823105 22208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72977e0a06114142b97bc12d107b2ef8, State: Initialized, Role: FOLLOWER
I20260620 14:50:27.823235 22208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72977e0a06114142b97bc12d107b2ef8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39371 } }
I20260620 14:50:27.823312 22208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:27.823333 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:27.820968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40889 (local address 127.0.98.62:39371)
0620 14:50:27.821136 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:27.821143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:27.821167 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:50:27.822034 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:27.822041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:27.822067 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:27.822184 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:27.822791 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:27.822794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:27.822794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:27.822798 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:27.822833 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:27.822843 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:27.822901 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:27.822970 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:50:27.823002 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260620 14:50:27.823335 22208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:27.823432 22208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:27.823966 22208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72977e0a06114142b97bc12d107b2ef8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39371 } }
I20260620 14:50:27.824061 22208 leader_election.cc:302] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72977e0a06114142b97bc12d107b2ef8; no voters:
I20260620 14:50:27.824196 22208 leader_election.cc:288] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:27.824193 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:27.824226 22213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:27.824378 22213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8 [term 1 LEADER]: Becoming Leader. State: Replica: 72977e0a06114142b97bc12d107b2ef8, State: Running, Role: LEADER
I20260620 14:50:27.824420 22208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:27.824527 22213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72977e0a06114142b97bc12d107b2ef8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39371 } }
I20260620 14:50:27.824847 22215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72977e0a06114142b97bc12d107b2ef8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72977e0a06114142b97bc12d107b2ef8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39371 } } }
I20260620 14:50:27.824970 22215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:27.825384 22222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:27.825649 22214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72977e0a06114142b97bc12d107b2ef8. Latest consensus state: current_term: 1 leader_uuid: "72977e0a06114142b97bc12d107b2ef8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72977e0a06114142b97bc12d107b2ef8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39371 } } }
I20260620 14:50:27.825838 22214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:27.825955 22222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:27.827307 22222 catalog_manager.cc:1347] Generated new cluster ID: 2ed691571c1e4857b8d463c0c32ae179
I20260620 14:50:27.827353 22222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:27.827936 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:27.825671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35217 (local address 127.0.98.62:39371)
0620 14:50:27.825704 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:27.825705 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:27.827020 (+ 1315us) server_negotiation.cc:407] Connection header received
0620 14:50:27.827044 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:27.827046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:27.827050 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:27.827113 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:27.827739 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:27.827740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:27.827741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:27.827745 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:27.827760 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:27.827764 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:27.827822 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:27.827864 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 14:50:27.827876 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:50:27.841331 22222 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:27.841508 22222 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:27.846189 22222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72977e0a06114142b97bc12d107b2ef8: Generated new TSK 0
I20260620 14:50:27.846359 22222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:27.855020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 22140
2026-06-20T14:50:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:27 dist-test-slave-cx89 krb5kdc[22239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:27 dist-test-slave-cx89 krb5kdc[22239](info): set up 2 sockets
Jun 20 14:50:27 dist-test-slave-cx89 krb5kdc[22239](info): commencing operation
krb5kdc: starting...
W20260620 14:50:28.811149 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:28 dist-test-slave-cx89 krb5kdc[22239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:50:28.921437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40705
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:29.000409 22252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:29.000609 22252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:29.000639 22252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:29.001745 22252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:29.001783 22252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:29.001802 22252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:29.001817 22252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:29.001834 22252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:29.003263 22252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:29.003506 22252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:29.003710 22252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:29.008240 22258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:29.008246 22260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:29.008279 22252 server_base.cc:1034] running on GCE node
W20260620 14:50:29.008246 22257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:29.008693 22252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:29.008971 22252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:29.010118 22252 hybrid_clock.cc:648] HybridClock initialized: now 1781967029010104 us; error 29 us; skew 500 ppm
Jun 20 14:50:29 dist-test-slave-cx89 krb5kdc[22239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:29.012566 22252 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:50:29.012964 22252 webserver.cc:460] Webserver started at http://127.0.98.62:37885/ using document root <none> and password file <none>
I20260620 14:50:29.013152 22252 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:29.013192 22252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:29.013275 22252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:29.014257 22252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "d96152c40def435f99b3435d5c6ef7f6"
format_stamp: "Formatted at 2026-06-20 14:50:29 on dist-test-slave-cx89"
server_key: "e1b87ba85a925a2e517673aa7a8c8faa"
server_key_iv: "a3046ac5f4878ce6ed762d1174108161"
server_key_version: "encryptionkey@0"
I20260620 14:50:29.014539 22252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "d96152c40def435f99b3435d5c6ef7f6"
format_stamp: "Formatted at 2026-06-20 14:50:29 on dist-test-slave-cx89"
server_key: "e1b87ba85a925a2e517673aa7a8c8faa"
server_key_iv: "a3046ac5f4878ce6ed762d1174108161"
server_key_version: "encryptionkey@0"
I20260620 14:50:29.015748 22252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:50:29.016390 22267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:29.016554 22252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:50:29.016633 22252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "d96152c40def435f99b3435d5c6ef7f6"
format_stamp: "Formatted at 2026-06-20 14:50:29 on dist-test-slave-cx89"
server_key: "e1b87ba85a925a2e517673aa7a8c8faa"
server_key_iv: "a3046ac5f4878ce6ed762d1174108161"
server_key_version: "encryptionkey@0"
I20260620 14:50:29.016685 22252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:29.029214 22252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:29.030875 22252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:29.030989 22252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:29.034559 22252 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40705
I20260620 14:50:29.034603 22319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40705 every 8 connection(s)
I20260620 14:50:29.034902 22252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:29.035669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 22252
I20260620 14:50:29.035763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:29.035776 22320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:29.035919 392 external_mini_cluster.cc:1428] Setting key cb92518270b870047b5c598050a6a580
I20260620 14:50:29.038908 22320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6: Bootstrap starting.
I20260620 14:50:29.039587 22320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:29.039849 22320 log.cc:824] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:29.040534 22320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6: No bootstrap required, opened a new log
Jun 20 14:50:29 dist-test-slave-cx89 krb5kdc[22239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:50:29.042409 22320 raft_consensus.cc:348] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96152c40def435f99b3435d5c6ef7f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } }
I20260620 14:50:29.042583 22320 raft_consensus.cc:374] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:29.042626 22320 raft_consensus.cc:729] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d96152c40def435f99b3435d5c6ef7f6, State: Initialized, Role: FOLLOWER
I20260620 14:50:29.042749 22320 consensus_queue.cc:260] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96152c40def435f99b3435d5c6ef7f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } }
I20260620 14:50:29.042809 22320 raft_consensus.cc:388] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:29.042837 22320 raft_consensus.cc:482] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:29.042884 22320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:29.043533 22320 raft_consensus.cc:504] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96152c40def435f99b3435d5c6ef7f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } }
I20260620 14:50:29.043623 22320 leader_election.cc:302] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d96152c40def435f99b3435d5c6ef7f6; no voters:
I20260620 14:50:29.043776 22320 leader_election.cc:288] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:29.043825 22325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:29.043979 22325 raft_consensus.cc:686] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6 [term 1 LEADER]: Becoming Leader. State: Replica: d96152c40def435f99b3435d5c6ef7f6, State: Running, Role: LEADER
I20260620 14:50:29.044027 22320 sys_catalog.cc:564] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:29.044153 22325 consensus_queue.cc:237] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96152c40def435f99b3435d5c6ef7f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } }
I20260620 14:50:29.044492 22326 sys_catalog.cc:455] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d96152c40def435f99b3435d5c6ef7f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96152c40def435f99b3435d5c6ef7f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } } }
I20260620 14:50:29.044559 22326 sys_catalog.cc:458] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:29.044705 22327 sys_catalog.cc:455] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d96152c40def435f99b3435d5c6ef7f6. Latest consensus state: current_term: 1 leader_uuid: "d96152c40def435f99b3435d5c6ef7f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96152c40def435f99b3435d5c6ef7f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40705 } } }
I20260620 14:50:29.044756 22327 sys_catalog.cc:458] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6 [sys.catalog]: This master's current role is: LEADER
W20260620 14:50:29.045732 22341 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:50:29.045825 22341 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:50:29.045910 22333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:29.046053 22333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:29.046216 22323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:29.036380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51513 (local address 127.0.98.62:40705)
0620 14:50:29.036577 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:29.036588 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:29.036620 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:50:29.037059 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:29.037068 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:29.037131 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:29.037282 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:29.037855 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:29.039203 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:29.040028 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:29.040526 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:29.040638 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:29.042884 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:29.042889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:29.042889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:29.042901 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:29.044755 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:29.045140 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:29.045143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:29.045144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:29.045230 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:29.045413 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:29.045414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:29.045414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:29.045667 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:29.045819 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:29.045899 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:50:29.046028 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
I20260620 14:50:29.047077 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:29.047545 22333 catalog_manager.cc:1347] Generated new cluster ID: 1aa7b42596ee4c39bff54bbee9890579
I20260620 14:50:29.047585 22333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:29.056929 22323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:29.051549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54927 (local address 127.0.98.62:40705)
0620 14:50:29.051599 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:29.051601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:29.051610 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:50:29.051631 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:29.051631 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:29.051640 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:29.051702 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:29.052249 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:29.052724 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:29.053390 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:29.053606 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:29.053660 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:29.054715 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:29.054717 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:29.054717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:29.054724 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:29.055974 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:29.056263 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:29.056264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:29.056264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:29.056315 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:29.056468 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:29.056469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:29.056469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:29.056571 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:29.056650 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:29.056772 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 14:50:29.056841 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:50:29.061690 22333 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:29.062239 22333 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:29.067044 22333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d96152c40def435f99b3435d5c6ef7f6: Generated new TSK 0
I20260620 14:50:29.067241 22333 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:29.072255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 22252
2026-06-20T14:50:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1217 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:29 dist-test-slave-cx89 krb5kdc[22351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:29 dist-test-slave-cx89 krb5kdc[22351](info): set up 2 sockets
Jun 20 14:50:29 dist-test-slave-cx89 krb5kdc[22351](info): commencing operation
krb5kdc: starting...
W20260620 14:50:30.023309 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:30 dist-test-slave-cx89 krb5kdc[22351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:50:30.130164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44553
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:30.206740 22364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:30.206945 22364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:30.206977 22364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:30.208128 22364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:30.208168 22364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:30.208186 22364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:30.208201 22364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:30.208216 22364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:30.209657 22364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:30.209894 22364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:30.210093 22364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:30.214490 22370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:30.214629 22364 server_base.cc:1034] running on GCE node
W20260620 14:50:30.214489 22372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:30.214510 22369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:30.214955 22364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:30.215225 22364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:30.216359 22364 hybrid_clock.cc:648] HybridClock initialized: now 1781967030216348 us; error 31 us; skew 500 ppm
Jun 20 14:50:30 dist-test-slave-cx89 krb5kdc[22351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:30.221925 22364 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:50:30.222379 22364 webserver.cc:460] Webserver started at http://127.0.98.62:37801/ using document root <none> and password file <none>
I20260620 14:50:30.222575 22364 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:30.222622 22364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:30.222736 22364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:30.223855 22364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "a6086ad5f66a40cebcbf1d2c2c1ccc68"
format_stamp: "Formatted at 2026-06-20 14:50:30 on dist-test-slave-cx89"
server_key: "44f6f4f1e361499ef8071ab999876e64"
server_key_iv: "0896ccba6efc1e5a01cedee3debf8513"
server_key_version: "encryptionkey@0"
I20260620 14:50:30.224195 22364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "a6086ad5f66a40cebcbf1d2c2c1ccc68"
format_stamp: "Formatted at 2026-06-20 14:50:30 on dist-test-slave-cx89"
server_key: "44f6f4f1e361499ef8071ab999876e64"
server_key_iv: "0896ccba6efc1e5a01cedee3debf8513"
server_key_version: "encryptionkey@0"
I20260620 14:50:30.225421 22364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:50:30.226130 22379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:30.226295 22364 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:30.226375 22364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "a6086ad5f66a40cebcbf1d2c2c1ccc68"
format_stamp: "Formatted at 2026-06-20 14:50:30 on dist-test-slave-cx89"
server_key: "44f6f4f1e361499ef8071ab999876e64"
server_key_iv: "0896ccba6efc1e5a01cedee3debf8513"
server_key_version: "encryptionkey@0"
I20260620 14:50:30.226434 22364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:30.247956 22364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:30.250029 22364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:30.250172 22364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:30.253728 22364 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44553
I20260620 14:50:30.253774 22431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44553 every 8 connection(s)
I20260620 14:50:30.254145 22364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:30.255183 22432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:30.257366 22432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68: Bootstrap starting.
I20260620 14:50:30.257877 22432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:30.258100 22432 log.cc:824] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:30.258714 22432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68: No bootstrap required, opened a new log
I20260620 14:50:30.260274 22432 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6086ad5f66a40cebcbf1d2c2c1ccc68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44553 } }
I20260620 14:50:30.260408 22432 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:30.260428 22432 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6086ad5f66a40cebcbf1d2c2c1ccc68, State: Initialized, Role: FOLLOWER
I20260620 14:50:30.260542 22432 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6086ad5f66a40cebcbf1d2c2c1ccc68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44553 } }
I20260620 14:50:30.260610 22432 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:30.260632 22432 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:30.260663 22432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:30.261107 22432 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6086ad5f66a40cebcbf1d2c2c1ccc68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44553 } }
I20260620 14:50:30.261186 22432 leader_election.cc:302] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6086ad5f66a40cebcbf1d2c2c1ccc68; no voters:
I20260620 14:50:30.261300 22432 leader_election.cc:288] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:30.261374 22435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:30.261521 22435 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68 [term 1 LEADER]: Becoming Leader. State: Replica: a6086ad5f66a40cebcbf1d2c2c1ccc68, State: Running, Role: LEADER
I20260620 14:50:30.261557 22432 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:30.261623 22435 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6086ad5f66a40cebcbf1d2c2c1ccc68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44553 } }
I20260620 14:50:30.261967 22436 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6086ad5f66a40cebcbf1d2c2c1ccc68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6086ad5f66a40cebcbf1d2c2c1ccc68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44553 } } }
I20260620 14:50:30.262042 22436 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:30.262094 22437 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6086ad5f66a40cebcbf1d2c2c1ccc68. Latest consensus state: current_term: 1 leader_uuid: "a6086ad5f66a40cebcbf1d2c2c1ccc68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6086ad5f66a40cebcbf1d2c2c1ccc68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44553 } } }
I20260620 14:50:30.262158 22437 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:30.262318 22443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:30.262848 22443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:30.264288 22443 catalog_manager.cc:1347] Generated new cluster ID: 62411975d6d744a2a4b04f766523a4c3
I20260620 14:50:30.264343 22443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:30.264396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 22364
I20260620 14:50:30.264497 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:30.264643 392 external_mini_cluster.cc:1428] Setting key 6edcdedbc94b63b4d22d3093b3ad444e
I20260620 14:50:30.275799 22455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:30.265048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39483 (local address 127.0.98.62:44553)
0620 14:50:30.265182 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:30.265190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:30.271772 (+ 6582us) server_negotiation.cc:407] Connection header received
0620 14:50:30.271903 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:30.271909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:30.271968 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:30.272115 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:30.272320 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:30.273247 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:30.274001 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:30.274408 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:30.274497 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:30.275116 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:30.275120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:30.275120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:30.275307 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:30.275338 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:30.275349 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:30.275400 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:30.275517 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 14:50:30.275584 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":86,"thread_start_us":64,"threads_started":1}
I20260620 14:50:30.276618 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:30.286468 22455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:30.283748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32937 (local address 127.0.98.62:44553)
0620 14:50:30.283790 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:30.283792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:30.283988 (+ 196us) server_negotiation.cc:407] Connection header received
0620 14:50:30.284043 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:30.284044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:30.284055 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:30.284099 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:30.284224 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:30.284709 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:30.285384 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:30.285598 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:30.285651 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:30.286244 (+ 593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:30.286246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:30.286246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:30.286251 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:30.286265 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:30.286269 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:30.286313 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:30.286362 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 14:50:30.286388 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:50:30.299625 22443 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:30.300344 22443 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:30.312150 22443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6086ad5f66a40cebcbf1d2c2c1ccc68: Generated new TSK 0
I20260620 14:50:30.312371 22443 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:30.322326 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 22364
2026-06-20T14:50:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1250 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:30 dist-test-slave-cx89 krb5kdc[22463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:30 dist-test-slave-cx89 krb5kdc[22463](info): set up 2 sockets
Jun 20 14:50:30 dist-test-slave-cx89 krb5kdc[22463](info): commencing operation
krb5kdc: starting...
W20260620 14:50:31.267262 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:31 dist-test-slave-cx89 krb5kdc[22463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:50:31.375573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41061
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:31.453037 22476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:31.453240 22476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:31.453261 22476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:31.454324 22476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:31.454356 22476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:31.454368 22476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:31.454378 22476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:31.454389 22476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:31.455812 22476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:31.455998 22476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:31.456164 22476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:31.460917 22481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:31.460923 22482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:31.460923 22484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:31.460985 22476 server_base.cc:1034] running on GCE node
I20260620 14:50:31.461359 22476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:31.461642 22476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:31.462765 22476 hybrid_clock.cc:648] HybridClock initialized: now 1781967031462758 us; error 28 us; skew 500 ppm
Jun 20 14:50:31 dist-test-slave-cx89 krb5kdc[22463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:31.465008 22476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:50:31.465408 22476 webserver.cc:460] Webserver started at http://127.0.98.62:35081/ using document root <none> and password file <none>
I20260620 14:50:31.465575 22476 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:31.465615 22476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:31.465703 22476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:31.466645 22476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "a6cbc47ab7184190a3afca81984063e9"
format_stamp: "Formatted at 2026-06-20 14:50:31 on dist-test-slave-cx89"
server_key: "ad317ff8a51aa2e8bc26383bc53c0a75"
server_key_iv: "6ff7e172436e318a0e3c48e13591348f"
server_key_version: "encryptionkey@0"
I20260620 14:50:31.466920 22476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "a6cbc47ab7184190a3afca81984063e9"
format_stamp: "Formatted at 2026-06-20 14:50:31 on dist-test-slave-cx89"
server_key: "ad317ff8a51aa2e8bc26383bc53c0a75"
server_key_iv: "6ff7e172436e318a0e3c48e13591348f"
server_key_version: "encryptionkey@0"
I20260620 14:50:31.468094 22476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:50:31.468720 22491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:31.468860 22476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:50:31.468937 22476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "a6cbc47ab7184190a3afca81984063e9"
format_stamp: "Formatted at 2026-06-20 14:50:31 on dist-test-slave-cx89"
server_key: "ad317ff8a51aa2e8bc26383bc53c0a75"
server_key_iv: "6ff7e172436e318a0e3c48e13591348f"
server_key_version: "encryptionkey@0"
I20260620 14:50:31.468997 22476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:31.472095 22476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:31.473702 22476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:31.473824 22476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:31.477372 22476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41061
I20260620 14:50:31.477411 22543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41061 every 8 connection(s)
I20260620 14:50:31.477691 22476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:31.478576 22544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:31.479527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 22476
I20260620 14:50:31.479622 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:31.479789 392 external_mini_cluster.cc:1428] Setting key 871b55d28f3088c2960c1211ef16205f
I20260620 14:50:31.481251 22544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9: Bootstrap starting.
I20260620 14:50:31.481874 22544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:31.482115 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:31.480238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47247 (local address 127.0.98.62:41061)
0620 14:50:31.480416 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:31.480426 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:31.480460 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:50:31.480978 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:31.480986 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:31.481029 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:31.481172 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:31.481718 (+ 546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:31.481721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:31.481722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:31.481725 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:31.481767 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:31.481780 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:31.481862 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:31.481901 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 14:50:31.481938 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260620 14:50:31.482208 22544 log.cc:824] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:31.482872 22544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9: No bootstrap required, opened a new log
I20260620 14:50:31.483003 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:31.484889 22544 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6cbc47ab7184190a3afca81984063e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41061 } }
I20260620 14:50:31.485061 22544 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:31.485101 22544 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6cbc47ab7184190a3afca81984063e9, State: Initialized, Role: FOLLOWER
I20260620 14:50:31.485189 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:31.484276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59013 (local address 127.0.98.62:41061)
0620 14:50:31.484316 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:31.484318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:31.484385 (+ 67us) server_negotiation.cc:407] Connection header received
0620 14:50:31.484423 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:31.484424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:31.484428 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:31.484496 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:31.484981 (+ 485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:31.484983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:31.484983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:50:31.484987 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:31.484999 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:50:31.485003 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:31.485062 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:31.485126 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 14:50:31.485137 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:50:31.485299 22544 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6cbc47ab7184190a3afca81984063e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41061 } }
I20260620 14:50:31.485388 22544 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:31.485425 22544 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:31.485469 22544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:31.485948 22544 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6cbc47ab7184190a3afca81984063e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41061 } }
I20260620 14:50:31.486035 22544 leader_election.cc:302] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6cbc47ab7184190a3afca81984063e9; no voters:
I20260620 14:50:31.486182 22544 leader_election.cc:288] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:31.486218 22554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:31.486342 22554 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9 [term 1 LEADER]: Becoming Leader. State: Replica: a6cbc47ab7184190a3afca81984063e9, State: Running, Role: LEADER
I20260620 14:50:31.486400 22544 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:31.486426 22554 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6cbc47ab7184190a3afca81984063e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41061 } }
I20260620 14:50:31.486750 22555 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6cbc47ab7184190a3afca81984063e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6cbc47ab7184190a3afca81984063e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41061 } } }
I20260620 14:50:31.486876 22555 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:31.486991 22556 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6cbc47ab7184190a3afca81984063e9. Latest consensus state: current_term: 1 leader_uuid: "a6cbc47ab7184190a3afca81984063e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6cbc47ab7184190a3afca81984063e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41061 } } }
I20260620 14:50:31.487048 22556 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:31.487203 22560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:31.487635 22560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:31.489141 22560 catalog_manager.cc:1347] Generated new cluster ID: 799ca7257a1e4b7a96f350fd10f02c1f
I20260620 14:50:31.489197 22560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:31.514097 22560 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:31.514302 22560 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:31.522171 22560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6cbc47ab7184190a3afca81984063e9: Generated new TSK 0
I20260620 14:50:31.522372 22560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:31.525179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 22476
2026-06-20T14:50:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1202 ms)
[----------] 18 tests from AuthTokenIssuingTest (21878 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:31 dist-test-slave-cx89 krb5kdc[22575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:31 dist-test-slave-cx89 krb5kdc[22575](info): set up 2 sockets
Jun 20 14:50:31 dist-test-slave-cx89 krb5kdc[22575](info): commencing operation
krb5kdc: starting...
W20260620 14:50:32.463171 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:32 dist-test-slave-cx89 krb5kdc[22575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:50:32.573535 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37889
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37889,127.0.98.61:39905,127.0.98.60:44867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:32.650847 22588 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:50:32.651062 22588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:32.651077 22588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:32.651096 22588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:32.652307 22588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:32.652355 22588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:32.652369 22588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:32.652379 22588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:32.652390 22588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:32.653919 22588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37889,127.0.98.61:39905,127.0.98.60:44867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:32.654147 22588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:32.654366 22588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:32.658926 22596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:32.658936 22593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:32.659102 22588 server_base.cc:1034] running on GCE node
W20260620 14:50:32.658910 22594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:32.659391 22588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:32.659694 22588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:32.660857 22588 hybrid_clock.cc:648] HybridClock initialized: now 1781967032660842 us; error 32 us; skew 500 ppm
Jun 20 14:50:32 dist-test-slave-cx89 krb5kdc[22575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:32.663228 22588 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:50:32.663667 22588 webserver.cc:460] Webserver started at http://127.0.98.62:37991/ using document root <none> and password file <none>
I20260620 14:50:32.663861 22588 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:32.663899 22588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:32.663988 22588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:32.664965 22588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "c912756204ed4e9f871ab026ebf30000"
format_stamp: "Formatted at 2026-06-20 14:50:32 on dist-test-slave-cx89"
server_key: "e20bea26edc041f512ba78f7f4020a9e"
server_key_iv: "ca5ddf9b4dfb4934e65c7585a003b5f5"
server_key_version: "encryptionkey@0"
I20260620 14:50:32.665244 22588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "c912756204ed4e9f871ab026ebf30000"
format_stamp: "Formatted at 2026-06-20 14:50:32 on dist-test-slave-cx89"
server_key: "e20bea26edc041f512ba78f7f4020a9e"
server_key_iv: "ca5ddf9b4dfb4934e65c7585a003b5f5"
server_key_version: "encryptionkey@0"
I20260620 14:50:32.666482 22588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:50:32.667112 22603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:32.667264 22588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:32.667342 22588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "c912756204ed4e9f871ab026ebf30000"
format_stamp: "Formatted at 2026-06-20 14:50:32 on dist-test-slave-cx89"
server_key: "e20bea26edc041f512ba78f7f4020a9e"
server_key_iv: "ca5ddf9b4dfb4934e65c7585a003b5f5"
server_key_version: "encryptionkey@0"
I20260620 14:50:32.667399 22588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:32.674993 22588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:32.676931 22588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:32.677043 22588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:32.680785 22588 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37889
I20260620 14:50:32.680850 22655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37889 every 8 connection(s)
I20260620 14:50:32.681140 22588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:32.682066 22656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:32.683619 22656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:32.687875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 22588
I20260620 14:50:32.687965 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:32.688139 392 external_mini_cluster.cc:1428] Setting key c821c00cc7ea6bdf389052ddde2820b4
Jun 20 14:50:32 dist-test-slave-cx89 krb5kdc[22575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:50:32.693229 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.684297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51206 (local address 127.0.98.62:37889)
0620 14:50:32.684490 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:32.684494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:32.684557 (+ 63us) server_negotiation.cc:407] Connection header received
0620 14:50:32.685103 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:32.685106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:32.685125 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:32.685199 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:32.685714 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.686496 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.687313 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.687775 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.687889 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.690092 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:32.690099 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:32.690099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:32.690105 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:32.691662 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.692090 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.692092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.692093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.692171 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.692425 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.692426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.692426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.692674 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:32.692765 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:32.692961 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 14:50:32.693067 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
I20260620 14:50:32.693293 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.684269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.0.1:51206)
0620 14:50:32.684432 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.684474 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:32.684955 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:32.685207 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:32.685209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:32.685472 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:32.685621 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.685625 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.686576 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.686577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.687238 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.687240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.687888 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.687889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.687955 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.688490 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:32.688499 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:32.689986 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:32.691746 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.691754 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.691757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.692011 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.692232 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.692233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.692233 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.692339 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.692752 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:32.692759 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:32.692838 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 14:50:32.692938 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:50:32.693160 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":62,"thread_start_us":79,"threads_started":1}
I20260620 14:50:32.694003 22656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:32.694334 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.694208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39905 (local address 127.0.0.1:52082)
0620 14:50:32.694257 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.694307 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 14:50:32.694495 22605 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39905: connect: Connection refused (error 111)
I20260620 14:50:32.694905 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.694812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39905 (local address 127.0.0.1:52084)
0620 14:50:32.694855 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.694884 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
W20260620 14:50:32.694999 22656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39905: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39905: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 14:50:32.710633 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39905
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37889,127.0.98.61:39905,127.0.98.60:44867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:50:32.744230 22656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } attempt: 1
I20260620 14:50:32.744769 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.744614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39905 (local address 127.0.0.1:52096)
0620 14:50:32.744674 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.744728 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:50:32.745117 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.745014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39905 (local address 127.0.0.1:52108)
0620 14:50:32.745060 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.745091 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 14:50:32.745241 22656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39905: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39905: connect: Connection refused (error 111)
W20260620 14:50:32.791158 22663 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:50:32.791374 22663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:32.791391 22663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:32.791467 22663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:32.792606 22663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:32.792647 22663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:32.792660 22663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:32.792671 22663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:32.792681 22663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:32.794131 22663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37889,127.0.98.61:39905,127.0.98.60:44867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:32.794358 22663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:32.794672 22663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:32.799506 22671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:32.799652 22669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:32.799583 22663 server_base.cc:1034] running on GCE node
W20260620 14:50:32.799566 22668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:32.799943 22663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:32.800235 22663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:32.801407 22663 hybrid_clock.cc:648] HybridClock initialized: now 1781967032801394 us; error 26 us; skew 500 ppm
Jun 20 14:50:32 dist-test-slave-cx89 krb5kdc[22575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:32.803802 22663 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 14:50:32.804217 22663 webserver.cc:460] Webserver started at http://127.0.98.61:33803/ using document root <none> and password file <none>
I20260620 14:50:32.804407 22663 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:32.804452 22663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:32.804574 22663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:32.805565 22663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-1/data/instance:
uuid: "80ad2333cde740cca3747a76ab34bd1d"
format_stamp: "Formatted at 2026-06-20 14:50:32 on dist-test-slave-cx89"
server_key: "9448165a0d29969d8fe021589e455e22"
server_key_iv: "94c80fc382270473a5fa7cff5eb1a90d"
server_key_version: "encryptionkey@0"
I20260620 14:50:32.805886 22663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-1/wal/instance:
uuid: "80ad2333cde740cca3747a76ab34bd1d"
format_stamp: "Formatted at 2026-06-20 14:50:32 on dist-test-slave-cx89"
server_key: "9448165a0d29969d8fe021589e455e22"
server_key_iv: "94c80fc382270473a5fa7cff5eb1a90d"
server_key_version: "encryptionkey@0"
I20260620 14:50:32.807060 22663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:50:32.807758 22678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:32.807931 22663 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:32.808012 22663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-1/wal
uuid: "80ad2333cde740cca3747a76ab34bd1d"
format_stamp: "Formatted at 2026-06-20 14:50:32 on dist-test-slave-cx89"
server_key: "9448165a0d29969d8fe021589e455e22"
server_key_iv: "94c80fc382270473a5fa7cff5eb1a90d"
server_key_version: "encryptionkey@0"
I20260620 14:50:32.808068 22663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:32.813388 22656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } attempt: 2
I20260620 14:50:32.813776 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.813642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39905 (local address 127.0.0.1:52116)
0620 14:50:32.813697 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.813740 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 14:50:32.814057 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.813995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39905 (local address 127.0.0.1:52124)
0620 14:50:32.814024 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.814043 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 14:50:32.814136 22656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39905: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39905: connect: Connection refused (error 111)
I20260620 14:50:32.824407 22663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:32.826129 22663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:32.826243 22663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:32.830016 22663 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39905
I20260620 14:50:32.830050 22730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39905 every 8 connection(s)
I20260620 14:50:32.830356 22663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:50:32.831214 22731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:32.832680 22731 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:32.835037 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 22663
I20260620 14:50:32.835115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-1/wal/instance
I20260620 14:50:32.835270 392 external_mini_cluster.cc:1428] Setting key be623c702703bcb7a5ca0b72b46f7408
Jun 20 14:50:32 dist-test-slave-cx89 krb5kdc[22575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:50:32.840811 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.833295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51214 (local address 127.0.98.62:37889)
0620 14:50:32.833355 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:32.833357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:32.833527 (+ 170us) server_negotiation.cc:407] Connection header received
0620 14:50:32.833967 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:32.833968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:32.833979 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:32.834032 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:32.834590 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.835060 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.836098 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.836298 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.836355 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.838312 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:32.838314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:32.838314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:32.838322 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:32.839632 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.839975 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.839976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.839977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.840028 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.840231 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.840231 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.840232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.840337 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:32.840391 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:32.840633 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 14:50:32.840713 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:50:32.840960 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.833274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.0.1:51214)
0620 14:50:32.833414 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.833454 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:32.833892 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:32.834086 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:32.834090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:32.834325 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:32.834527 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.834531 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.835143 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.835145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.836007 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.836009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.836375 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.836376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.836493 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.836930 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:32.836938 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:32.838228 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:32.839692 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.839699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.839702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.839911 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.840078 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.840079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.840079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.840178 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.840387 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:32.840393 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:32.840523 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 14:50:32.840607 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:50:32.840751 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260620 14:50:32.841478 22731 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 14:50:32 dist-test-slave-cx89 krb5kdc[22575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:50:32.847935 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.841713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39905 (local address 127.0.0.1:52140)
0620 14:50:32.841870 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.841880 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:32.841926 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:32.842049 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:32.842050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:32.842174 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:32.842214 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.842216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.843096 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.843097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.843700 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.843702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.844124 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.844124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.844182 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.844550 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:32.844554 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:32.845671 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:32.847065 (+ 1394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.847066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.847066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.847209 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.847371 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.847372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.847372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.847448 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.847681 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:32.847682 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:32.847719 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:50:32.847783 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:50:32.847869 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260620 14:50:32.848048 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.841752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52140 (local address 127.0.98.61:39905)
0620 14:50:32.841886 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:32.841891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:32.841924 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:50:32.841974 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:32.841976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:32.841989 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:32.842050 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:32.842273 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.843033 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.843759 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.844065 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.844122 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.845739 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:32.845746 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:32.845747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:32.845753 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:32.846999 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.847271 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.847274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.847274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.847323 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.847490 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.847491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.847491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.847627 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:32.847696 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:32.847863 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 14:50:32.847948 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260620 14:50:32.848579 22731 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:32.848912 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.848781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44867 (local address 127.0.0.1:40708)
0620 14:50:32.848835 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.848887 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 14:50:32.849049 22681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44867: connect: Connection refused (error 111)
I20260620 14:50:32.849424 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.849335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44867 (local address 127.0.0.1:40722)
0620 14:50:32.849375 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.849403 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 14:50:32.849520 22731 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44867: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44867: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 14:50:32.858151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44867
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37889,127.0.98.61:39905,127.0.98.60:44867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:50:32.898728 22731 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } attempt: 1
I20260620 14:50:32.899209 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.899086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44867 (local address 127.0.0.1:40730)
0620 14:50:32.899144 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.899182 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:50:32.899549 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.899480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44867 (local address 127.0.0.1:40744)
0620 14:50:32.899511 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.899533 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260620 14:50:32.899634 22731 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44867: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44867: connect: Connection refused (error 111)
I20260620 14:50:32.905262 22656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } attempt: 3
Jun 20 14:50:32 dist-test-slave-cx89 krb5kdc[22575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:50:32.912681 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.905590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.0.98.61:39905)
0620 14:50:32.905635 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:32.905637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:32.905652 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:50:32.905725 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:32.905726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:32.905739 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:32.905819 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:32.906086 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.906852 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.907889 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.908203 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.908260 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.910218 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:32.910220 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:32.910221 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:32.910228 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:32.911692 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.912052 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.912053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.912053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.912106 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.912279 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.912280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.912280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.912379 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:32.912426 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:32.912560 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 14:50:32.912621 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:50:32.912734 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.905542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39905 (local address 127.0.0.1:52156)
0620 14:50:32.905595 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.905606 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:32.905663 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:32.905826 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:32.905827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:32.905964 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:32.906026 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.906028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.906955 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.906956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.907799 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.907803 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.908309 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.908310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.908422 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.908923 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:32.908929 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:32.910146 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:32.911778 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.911779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.911780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.911992 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.912164 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.912165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.912165 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.912227 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.912428 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:32.912429 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:32.912477 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:50:32.912543 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:50:32.912636 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:50:32.913074 22656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:32.913384 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.913288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44867 (local address 127.0.0.1:40756)
0620 14:50:32.913328 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.913359 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 14:50:32.913673 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.913598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44867 (local address 127.0.0.1:40760)
0620 14:50:32.913634 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.913657 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 14:50:32.913760 22656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44867: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44867: connect: Connection refused (error 111)
W20260620 14:50:32.938199 22738 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:50:32.938406 22738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:32.938421 22738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:32.938441 22738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:32.939594 22738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:32.939633 22738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:32.939646 22738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:32.939658 22738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:32.939682 22738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:32.941133 22738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37889,127.0.98.61:39905,127.0.98.60:44867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:32.941362 22738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:32.941566 22738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:50:32.944898 22656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } attempt: 1
I20260620 14:50:32.945303 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.945164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44867 (local address 127.0.0.1:40772)
0620 14:50:32.945227 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.945266 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 14:50:32.945734 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.945630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44867 (local address 127.0.0.1:40784)
0620 14:50:32.945686 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.945714 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 14:50:32.945868 22656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44867: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44867: connect: Connection refused (error 111)
W20260620 14:50:32.946036 22744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:32.946027 22746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:32.946038 22743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:32.946408 22738 server_base.cc:1034] running on GCE node
I20260620 14:50:32.946616 22738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:32.946867 22738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:32.947997 22738 hybrid_clock.cc:648] HybridClock initialized: now 1781967032947982 us; error 26 us; skew 500 ppm
Jun 20 14:50:32 dist-test-slave-cx89 krb5kdc[22575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:32.950287 22738 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 14:50:32.950712 22738 webserver.cc:460] Webserver started at http://127.0.98.60:46479/ using document root <none> and password file <none>
I20260620 14:50:32.950917 22738 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:32.950963 22738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:32.951079 22738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:32.952111 22738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-2/data/instance:
uuid: "cf1525599a004989b3c5dc19493be629"
format_stamp: "Formatted at 2026-06-20 14:50:32 on dist-test-slave-cx89"
server_key: "067aeec566186fb4fc44abe210581638"
server_key_iv: "42dc3eeb0f59f57f2b6f017e9761e668"
server_key_version: "encryptionkey@0"
I20260620 14:50:32.952425 22738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-2/wal/instance:
uuid: "cf1525599a004989b3c5dc19493be629"
format_stamp: "Formatted at 2026-06-20 14:50:32 on dist-test-slave-cx89"
server_key: "067aeec566186fb4fc44abe210581638"
server_key_iv: "42dc3eeb0f59f57f2b6f017e9761e668"
server_key_version: "encryptionkey@0"
I20260620 14:50:32.953596 22738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:50:32.954196 22753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:32.954384 22738 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:32.954466 22738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-2/wal
uuid: "cf1525599a004989b3c5dc19493be629"
format_stamp: "Formatted at 2026-06-20 14:50:32 on dist-test-slave-cx89"
server_key: "067aeec566186fb4fc44abe210581638"
server_key_iv: "42dc3eeb0f59f57f2b6f017e9761e668"
server_key_version: "encryptionkey@0"
I20260620 14:50:32.954535 22738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:32.966266 22738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:32.967804 22731 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } attempt: 2
I20260620 14:50:32.968117 22738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:32.968165 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.968042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44867 (local address 127.0.0.1:40786)
0620 14:50:32.968096 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.968131 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 14:50:32.968236 22738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:32.968493 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.968408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44867 (local address 127.0.0.1:40790)
0620 14:50:32.968443 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.968468 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 14:50:32.968582 22731 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44867: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44867: connect: Connection refused (error 111)
I20260620 14:50:32.971956 22738 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44867
I20260620 14:50:32.971992 22805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44867 every 8 connection(s)
I20260620 14:50:32.972298 22738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:50:32.973147 22806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:32.974596 22806 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:50:32 dist-test-slave-cx89 krb5kdc[22575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:50:32.982170 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.975190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51226 (local address 127.0.98.62:37889)
0620 14:50:32.975288 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:32.975291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:32.975471 (+ 180us) server_negotiation.cc:407] Connection header received
0620 14:50:32.975945 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:32.975946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:32.975955 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:32.975994 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:32.976546 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.976995 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.977893 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.978061 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.978104 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.980055 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:32.980056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:32.980056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:32.980062 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:32.981230 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.981520 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.981520 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.981521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.981564 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.981723 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.981723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.981724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.981818 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:32.981860 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:32.982054 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 14:50:32.982107 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:50:32.982342 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.975167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.0.1:51226)
0620 14:50:32.975346 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.975381 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:32.975854 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:32.976057 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:32.976060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:32.976303 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:32.976486 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.976490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.977056 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.977057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.977836 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.977838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.978120 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.978121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.978222 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.978696 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:32.978705 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:32.979976 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:32.981290 (+ 1314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.981293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.981294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.981471 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.981604 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.981605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.981605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.981685 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.981858 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:32.981860 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:32.981965 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 14:50:32.982037 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:50:32.982142 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260620 14:50:32.982339 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 22738
I20260620 14:50:32.982414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966748446763-392-0/minicluster-data/master-2/wal/instance
I20260620 14:50:32.982568 392 external_mini_cluster.cc:1428] Setting key 2c50c4ef4c32459ed66e81c83a723c12
I20260620 14:50:32.982968 22806 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:50:32 dist-test-slave-cx89 krb5kdc[22575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 14:50:32 dist-test-slave-cx89 krb5kdc[22575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:50:32.988906 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.983242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39905 (local address 127.0.0.1:52168)
0620 14:50:32.983278 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.983286 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:32.983314 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:32.983485 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:32.983486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:32.983607 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:32.983652 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.983653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.984216 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.984217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.984773 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.984774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.985102 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.985102 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.985163 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.985505 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:32.985509 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:32.986727 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:32.988054 (+ 1327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.988055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.988055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.988201 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.988365 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.988366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.988366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.988419 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.988629 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:32.988630 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:32.988670 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:50:32.988729 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:50:32.988810 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":65}
I20260620 14:50:32.988911 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.983315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52168 (local address 127.0.98.61:39905)
0620 14:50:32.983364 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:32.983366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:32.983374 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:50:32.983394 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:32.983395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:32.983403 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:32.983473 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:32.983713 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.984157 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.984840 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.985049 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.985101 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.986812 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:32.986813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:32.986813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:32.986820 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:32.987984 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.988259 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.988261 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.988261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.988315 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.988468 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.988469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.988469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.988578 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:32.988630 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:32.988765 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 14:50:32.988829 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:50:32.989197 22806 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:32.990305 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.982944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40798 (local address 127.0.98.60:44867)
0620 14:50:32.983072 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:32.983076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:32.983108 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:50:32.983172 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:32.983175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:32.983188 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:32.983255 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:32.983857 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.984521 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.985207 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.985610 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.985666 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.987841 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:32.987847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:32.987848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:32.987854 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:32.989255 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.989577 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.989580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.989580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.989638 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.989807 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.989808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.989808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.989951 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:32.990095 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:32.990134 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 14:50:32.990202 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
Jun 20 14:50:32 dist-test-slave-cx89 krb5kdc[22575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 14:50:32 dist-test-slave-cx89 krb5kdc[22575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:50:32.996325 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.989405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44867 (local address 127.0.0.1:40802)
0620 14:50:32.989454 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.989464 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:32.989501 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:32.989658 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:32.989659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:32.989780 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:32.989814 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.989815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.990624 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.990625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.991269 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.991271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.991730 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.991731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.991780 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.992066 (+ 286us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:32.992070 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:32.993287 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:32.995371 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.995372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.995372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.995573 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.995756 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.995757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.995757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.995814 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.996040 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:32.996041 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:32.996076 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:50:32.996137 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:50:32.996224 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:50:32.996407 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.989435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40802 (local address 127.0.98.60:44867)
0620 14:50:32.989555 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:32.989556 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:32.989562 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:50:32.989577 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:32.989578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:32.989586 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:32.989656 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:32.989876 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.990545 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.991336 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.991675 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.991719 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.993353 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:32.993354 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:32.993355 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:32.993360 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:32.995291 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.995634 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.995634 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.995635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.995692 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.995865 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.995866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.995866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.995987 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:32.996049 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:32.996174 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:50:32.996271 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
I20260620 14:50:32.997761 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.991007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51240 (local address 127.0.98.62:37889)
0620 14:50:32.991041 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:32.991043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:32.991050 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:50:32.991070 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:32.991071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:32.991079 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:32.991137 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:32.991594 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.992202 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.993026 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.993261 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.993328 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.995338 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:32.995339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:32.995339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:32.995345 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:32.996831 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.997150 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.997151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.997151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.997199 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.997375 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.997375 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.997375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.997456 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:32.997555 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:32.997631 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 14:50:32.997705 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:50:32.998101 22806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629: Bootstrap starting.
I20260620 14:50:32.998842 22806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:32.999126 22806 log.cc:824] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:32.999872 22806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629: No bootstrap required, opened a new log
I20260620 14:50:33.001762 22806 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c912756204ed4e9f871ab026ebf30000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } peers { permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } } peers { permanent_uuid: "cf1525599a004989b3c5dc19493be629" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } }
I20260620 14:50:33.001917 22806 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1525599a004989b3c5dc19493be629, State: Initialized, Role: FOLLOWER
I20260620 14:50:33.001996 22806 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c912756204ed4e9f871ab026ebf30000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } peers { permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } } peers { permanent_uuid: "cf1525599a004989b3c5dc19493be629" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } }
I20260620 14:50:33.002259 22813 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c912756204ed4e9f871ab026ebf30000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } peers { permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } } peers { permanent_uuid: "cf1525599a004989b3c5dc19493be629" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } } }
I20260620 14:50:33.002303 22806 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:33.002353 22813 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 14:50:32 dist-test-slave-cx89 krb5kdc[22575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260620 14:50:33.004088 22824 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:50:33.004189 22824 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:50:33.005966 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.998326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.0.98.61:39905)
0620 14:50:32.998360 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:32.998362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:32.998404 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:50:32.998473 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:32.998473 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:32.998481 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:32.998518 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:32.999031 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.999570 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.000418 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.000615 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.000693 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.003063 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:33.003064 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:33.003065 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:33.003072 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:33.005049 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.005340 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.005342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.005342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.005392 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.005546 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.005547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.005547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.005651 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:33.005787 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:33.005821 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 14:50:33.005882 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:50:33.004464 22738 master_runner.cc:186] Error getting master registration for 127.0.98.62:37889: OK, instance_id { permanent_uuid: "c912756204ed4e9f871ab026ebf30000" instance_seqno: 1781967032679785 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:50:33.006531 22738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:50:33.007234 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:33.008297 22630 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c912756204ed4e9f871ab026ebf30000"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51240
I20260620 14:50:33.021008 22656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } attempt: 2
Jun 20 14:50:33 dist-test-slave-cx89 krb5kdc[22575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:50:33.028527 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.021330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44867 (local address 127.0.0.1:40818)
0620 14:50:33.021372 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.021386 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:33.021445 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:33.021579 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:33.021580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:33.021760 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:33.021812 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:33.021814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.022734 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.022735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:33.023565 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:33.023567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.024037 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.024038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:33.024091 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.024503 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:33.024509 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:33.025835 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:33.027380 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:33.027381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:33.027382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:33.027649 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:33.027839 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:33.027840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:33.027840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:33.027903 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:33.028208 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:33.028209 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:33.028264 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 14:50:33.028331 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:50:33.028424 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 14:50:33.028538 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.021387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40818 (local address 127.0.98.60:44867)
0620 14:50:33.021436 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.021438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.021448 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:50:33.021499 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.021500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.021510 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.021559 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.021884 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.022665 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.023647 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.023971 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.024036 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.025898 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:33.025917 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:33.025917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:33.025925 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:33.027189 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.027718 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.027720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.027720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.027775 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.027959 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.027960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.027960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.028151 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:33.028212 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:33.028358 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 14:50:33.028443 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:50:33.029817 22656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000: Bootstrap starting.
I20260620 14:50:33.030402 22656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:33.030651 22656 log.cc:824] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:33.031488 22656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000: No bootstrap required, opened a new log
I20260620 14:50:33.032819 22656 raft_consensus.cc:348] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c912756204ed4e9f871ab026ebf30000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } peers { permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } } peers { permanent_uuid: "cf1525599a004989b3c5dc19493be629" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } }
I20260620 14:50:33.032965 22656 raft_consensus.cc:729] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c912756204ed4e9f871ab026ebf30000, State: Initialized, Role: FOLLOWER
I20260620 14:50:33.033054 22656 consensus_queue.cc:260] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c912756204ed4e9f871ab026ebf30000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } peers { permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } } peers { permanent_uuid: "cf1525599a004989b3c5dc19493be629" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } }
I20260620 14:50:33.033324 22827 sys_catalog.cc:455] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c912756204ed4e9f871ab026ebf30000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } peers { permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } } peers { permanent_uuid: "cf1525599a004989b3c5dc19493be629" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } } }
I20260620 14:50:33.033416 22630 raft_consensus.cc:482] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:50:33.033375 22656 sys_catalog.cc:564] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:33.033476 22630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:33.034094 22630 raft_consensus.cc:504] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c912756204ed4e9f871ab026ebf30000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } peers { permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } } peers { permanent_uuid: "cf1525599a004989b3c5dc19493be629" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } }
I20260620 14:50:33.034582 22780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c912756204ed4e9f871ab026ebf30000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cf1525599a004989b3c5dc19493be629"
I20260620 14:50:33.033447 22827 sys_catalog.cc:458] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:33.034718 22780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629 [term 0 FOLLOWER]: Advancing to term 1
W20260620 14:50:33.035370 22838 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:50:33.035494 22838 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:50:33.035593 22780 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c912756204ed4e9f871ab026ebf30000 in term 1.
I20260620 14:50:33.036034 22630 leader_election.cc:288] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [CANDIDATE]: Term 1 election: Requested vote from peers 80ad2333cde740cca3747a76ab34bd1d (127.0.98.61:39905), cf1525599a004989b3c5dc19493be629 (127.0.98.60:44867)
I20260620 14:50:33.036093 22606 leader_election.cc:302] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c912756204ed4e9f871ab026ebf30000, cf1525599a004989b3c5dc19493be629; no voters:
I20260620 14:50:33.036181 22827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:33.036301 22827 raft_consensus.cc:686] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [term 1 LEADER]: Becoming Leader. State: Replica: c912756204ed4e9f871ab026ebf30000, State: Running, Role: LEADER
I20260620 14:50:33.036377 22827 consensus_queue.cc:237] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c912756204ed4e9f871ab026ebf30000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } peers { permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } } peers { permanent_uuid: "cf1525599a004989b3c5dc19493be629" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } }
I20260620 14:50:33.036758 22827 sys_catalog.cc:455] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c912756204ed4e9f871ab026ebf30000. Latest consensus state: current_term: 1 leader_uuid: "c912756204ed4e9f871ab026ebf30000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c912756204ed4e9f871ab026ebf30000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } peers { permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } } peers { permanent_uuid: "cf1525599a004989b3c5dc19493be629" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } } }
I20260620 14:50:33.036832 22827 sys_catalog.cc:458] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:33.037555 22588 master_runner.cc:186] Error getting master registration for 127.0.98.61:39905: OK, instance_id { permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" instance_seqno: 1781967032828994 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:50:33.039777 22588 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:50:33.052184 22846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:33.052431 22846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:33.053393 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.043730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40824 (local address 127.0.98.60:44867)
0620 14:50:33.043774 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.043776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.044096 (+ 320us) server_negotiation.cc:407] Connection header received
0620 14:50:33.044219 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.044220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.044233 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.044287 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.044994 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.045735 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.047476 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.047782 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.047851 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.049863 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:33.049864 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:33.049864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:33.049871 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:33.051724 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.052313 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.052315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.052315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.052381 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.052666 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.052666 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.052667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.052830 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:33.052916 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:33.053219 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 14:50:33.053299 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:50:33.053395 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.044096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.0.98.61:39905)
0620 14:50:33.044136 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.044137 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.044281 (+ 144us) server_negotiation.cc:407] Connection header received
0620 14:50:33.045059 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.045060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.045070 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.045098 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.045756 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.046467 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.048293 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.048567 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.048622 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.050642 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:33.050643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:33.050644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:33.050651 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:33.052157 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.052622 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.052623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.052623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.052673 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.052854 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.052855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.052855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.052974 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:33.053045 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:33.053228 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 14:50:33.053311 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:50:33.059495 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.043953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51242 (local address 127.0.98.62:37889)
0620 14:50:33.043988 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.043989 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.044278 (+ 289us) server_negotiation.cc:407] Connection header received
0620 14:50:33.051561 (+ 7283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.051563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.051573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.051602 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.053332 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.054046 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.055040 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.055349 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.055429 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.056731 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:33.056732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:33.056732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:33.056739 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:33.058378 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.058755 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.058757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.058757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.058825 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.059003 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.059003 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.059004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.059106 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:33.059208 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:33.059311 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 14:50:33.059382 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 14:50:33.059690 22731 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } attempt: 3
I20260620 14:50:33.060999 22780 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629 [term 1 FOLLOWER]: Refusing update from remote peer c912756204ed4e9f871ab026ebf30000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:33.061316 22840 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf1525599a004989b3c5dc19493be629" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:33.063654 22813 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c912756204ed4e9f871ab026ebf30000. Latest consensus state: current_term: 1 leader_uuid: "c912756204ed4e9f871ab026ebf30000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c912756204ed4e9f871ab026ebf30000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } peers { permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } } peers { permanent_uuid: "cf1525599a004989b3c5dc19493be629" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } } }
I20260620 14:50:33.063756 22813 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 14:50:33 dist-test-slave-cx89 krb5kdc[22575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:50:33.065181 22840 sys_catalog.cc:455] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c912756204ed4e9f871ab026ebf30000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c912756204ed4e9f871ab026ebf30000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } peers { permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } } peers { permanent_uuid: "cf1525599a004989b3c5dc19493be629" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } } }
I20260620 14:50:33.065275 22840 sys_catalog.cc:458] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:33.065703 22813 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c912756204ed4e9f871ab026ebf30000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c912756204ed4e9f871ab026ebf30000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } peers { permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } } peers { permanent_uuid: "cf1525599a004989b3c5dc19493be629" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } } }
I20260620 14:50:33.065775 22813 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:33.066018 22850 mvcc.cc:205] Tried to move back new op lower bound from 7298936967416057856 to 7298936967318126592. Current Snapshot: MvccSnapshot[applied={T|T < 7298936967416057856}]
I20260620 14:50:33.066556 22851 mvcc.cc:205] Tried to move back new op lower bound from 7298936967416057856 to 7298936967318126592. Current Snapshot: MvccSnapshot[applied={T|T < 7298936967416057856}]
I20260620 14:50:33.068145 22846 catalog_manager.cc:1347] Generated new cluster ID: b48dc28d3f004752b61f98268086640d
I20260620 14:50:33.068199 22846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:33.068781 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.059925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44867 (local address 127.0.0.1:40838)
0620 14:50:33.059969 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.059981 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:33.060016 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:33.060138 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:33.060139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:33.060288 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:33.060328 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:33.060330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.061283 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.061284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:33.062092 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:33.062094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.062883 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.062884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:33.062951 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.063407 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:33.063433 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:33.064881 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:33.067511 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:33.067512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:33.067513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:33.067717 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:33.067902 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:33.067903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:33.067903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:33.067959 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:33.068479 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:33.068480 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:33.068542 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 14:50:33.068591 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:50:33.068686 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 14:50:33.069720 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.060008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.0.98.60:44867)
0620 14:50:33.060042 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.060044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.060051 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:50:33.060070 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.060071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.060079 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.060129 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.060397 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.060929 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.062169 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.062772 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.064970 (+ 2198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.065045 (+ 75us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:33.065046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:33.065046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:33.065078 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:33.067426 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.067783 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.067784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.067784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.067838 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.068013 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.068013 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.068014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.068344 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:33.069034 (+ 690us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:33.069102 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 14:50:33.069223 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:50:33.072888 22731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d: Bootstrap starting.
I20260620 14:50:33.073606 22731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:33.073881 22731 log.cc:824] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:33.079828 22731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d: No bootstrap required, opened a new log
I20260620 14:50:33.081779 22731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c912756204ed4e9f871ab026ebf30000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } peers { permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } } peers { permanent_uuid: "cf1525599a004989b3c5dc19493be629" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } }
I20260620 14:50:33.081915 22731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ad2333cde740cca3747a76ab34bd1d, State: Initialized, Role: FOLLOWER
I20260620 14:50:33.081993 22731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c912756204ed4e9f871ab026ebf30000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } peers { permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } } peers { permanent_uuid: "cf1525599a004989b3c5dc19493be629" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } }
I20260620 14:50:33.082274 22856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c912756204ed4e9f871ab026ebf30000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } peers { permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } } peers { permanent_uuid: "cf1525599a004989b3c5dc19493be629" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } } }
I20260620 14:50:33.082343 22731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:33.082430 22705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c912756204ed4e9f871ab026ebf30000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "80ad2333cde740cca3747a76ab34bd1d"
I20260620 14:50:33.082542 22704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:33.082617 22856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:33.083350 22704 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d [term 1 FOLLOWER]: Refusing update from remote peer c912756204ed4e9f871ab026ebf30000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:33.083655 22827 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:33.084120 22705 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c912756204ed4e9f871ab026ebf30000 in term 1.
I20260620 14:50:33.086539 22857 mvcc.cc:205] Tried to move back new op lower bound from 7298936967416057856 to 7298936967318126592. Current Snapshot: MvccSnapshot[applied={T|T < 7298936967416057856}]
I20260620 14:50:33.087145 22856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c912756204ed4e9f871ab026ebf30000. Latest consensus state: current_term: 1 leader_uuid: "c912756204ed4e9f871ab026ebf30000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c912756204ed4e9f871ab026ebf30000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } peers { permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } } peers { permanent_uuid: "cf1525599a004989b3c5dc19493be629" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } } }
I20260620 14:50:33.087237 22856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:33.087275 22856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c912756204ed4e9f871ab026ebf30000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c912756204ed4e9f871ab026ebf30000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } peers { permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } } peers { permanent_uuid: "cf1525599a004989b3c5dc19493be629" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } } }
I20260620 14:50:33.087308 22856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:33.088469 22871 catalog_manager.cc:1259] Loaded cluster ID: b48dc28d3f004752b61f98268086640d
I20260620 14:50:33.088506 22871 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d: loading cluster ID for follower catalog manager: success
W20260620 14:50:33.088587 22871 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260620 14:50:33.088618 22871 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:50:33.089416 22846 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:33.090199 22846 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:33.096177 22840 sys_catalog.cc:455] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c912756204ed4e9f871ab026ebf30000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c912756204ed4e9f871ab026ebf30000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } peers { permanent_uuid: "80ad2333cde740cca3747a76ab34bd1d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39905 } } peers { permanent_uuid: "cf1525599a004989b3c5dc19493be629" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44867 } } }
I20260620 14:50:33.096287 22840 sys_catalog.cc:458] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:33.098456 22846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c912756204ed4e9f871ab026ebf30000: Generated new TSK 0
I20260620 14:50:33.098691 22846 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:50:33.122711 22811 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:33.121082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40846 (local address 127.0.98.60:44867)
0620 14:50:33.121127 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.121129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.122059 (+ 930us) server_negotiation.cc:407] Connection header received
0620 14:50:33.122164 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.122165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.122176 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.122206 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.122640 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40846: BlockingRecv error: recv got EOF from 127.0.0.1:40846 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 14:50:33.123477 22659 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:33.121495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51252 (local address 127.0.98.62:37889)
0620 14:50:33.121536 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.121538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.122946 (+ 1408us) server_negotiation.cc:407] Connection header received
0620 14:50:33.123038 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.123039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.123050 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.123077 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.123393 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51252: BlockingRecv error: recv got EOF from 127.0.0.1:51252 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 14:50:33.124285 22735 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:33.121799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52188 (local address 127.0.98.61:39905)
0620 14:50:33.121838 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.121839 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.123666 (+ 1827us) server_negotiation.cc:407] Connection header received
0620 14:50:33.123825 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.123827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.123837 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.123866 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.124202 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52188: BlockingRecv error: recv got EOF from 127.0.0.1:52188 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 14:50:33.132568 22735 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:33.131693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.0.98.61:39905)
0620 14:50:33.131734 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.131736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.131989 (+ 253us) server_negotiation.cc:407] Connection header received
0620 14:50:33.132097 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.132098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.132119 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.132148 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.132514 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52202: BlockingRecv error: recv got EOF from 127.0.0.1:52202 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 14:50:33.135701 22735 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:33.134913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.0.98.61:39905)
0620 14:50:33.134947 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.134949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.135184 (+ 235us) server_negotiation.cc:407] Connection header received
0620 14:50:33.135286 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.135287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.135297 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.135324 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.135652 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52208: BlockingRecv error: recv got EOF from 127.0.0.1:52208 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 14:50:33.139536 22735 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:33.138778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52220 (local address 127.0.98.61:39905)
0620 14:50:33.138823 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.138825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.139017 (+ 192us) server_negotiation.cc:407] Connection header received
0620 14:50:33.139102 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.139103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.139113 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.139138 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.139488 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52220: BlockingRecv error: recv got EOF from 127.0.0.1:52220 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260620 14:50:33.145496 22735 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:33.144691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52228 (local address 127.0.98.61:39905)
0620 14:50:33.144728 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.144730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.144960 (+ 230us) server_negotiation.cc:407] Connection header received
0620 14:50:33.145052 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.145053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.145062 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.145088 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.145442 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52228: BlockingRecv error: recv got EOF from 127.0.0.1:52228 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 14:50:33.155861 22735 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:33.154862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52244 (local address 127.0.98.61:39905)
0620 14:50:33.154906 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.154907 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.155175 (+ 268us) server_negotiation.cc:407] Connection header received
0620 14:50:33.155276 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.155277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.155289 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.155318 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.155791 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52244: BlockingRecv error: recv got EOF from 127.0.0.1:52244 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 14:50:33.173781 22735 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:33.173127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52254 (local address 127.0.98.61:39905)
0620 14:50:33.173171 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.173173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.173264 (+ 91us) server_negotiation.cc:407] Connection header received
0620 14:50:33.173333 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.173334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.173345 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.173385 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.173719 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52254: BlockingRecv error: recv got EOF from 127.0.0.1:52254 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 14:50:33.207399 22735 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:33.206745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.0.98.61:39905)
0620 14:50:33.206791 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.206793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.206919 (+ 126us) server_negotiation.cc:407] Connection header received
0620 14:50:33.206984 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.206984 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.206996 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.207037 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.207338 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52262: BlockingRecv error: recv got EOF from 127.0.0.1:52262 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 14:50:33.273062 22735 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:33.272419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52278 (local address 127.0.98.61:39905)
0620 14:50:33.272469 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.272470 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.272557 (+ 87us) server_negotiation.cc:407] Connection header received
0620 14:50:33.272620 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.272621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.272630 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.272679 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.272998 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52278: BlockingRecv error: recv got EOF from 127.0.0.1:52278 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260620 14:50:33.402725 22735 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:33.402121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52292 (local address 127.0.98.61:39905)
0620 14:50:33.402187 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.402189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.402235 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:50:33.402290 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.402291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.402301 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.402365 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.402646 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52292: BlockingRecv error: recv got EOF from 127.0.0.1:52292 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 14:50:33.660308 22735 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:33.659782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.0.98.61:39905)
0620 14:50:33.659841 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.659842 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.659855 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:50:33.659901 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.659901 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.659912 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.659982 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.660258 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52302: BlockingRecv error: recv got EOF from 127.0.0.1:52302 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:50:34.004644 22824 catalog_manager.cc:1259] Loaded cluster ID: b48dc28d3f004752b61f98268086640d
I20260620 14:50:34.004717 22824 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629: loading cluster ID for follower catalog manager: success
I20260620 14:50:34.005525 22824 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629: acquiring CA information for follower catalog manager: success
I20260620 14:50:34.005810 22824 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cf1525599a004989b3c5dc19493be629: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:50:34.089699 22871 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d: acquiring CA information for follower catalog manager: success
I20260620 14:50:34.090055 22871 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 80ad2333cde740cca3747a76ab34bd1d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:50:34.175858 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:34.173385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.0.98.61:39905)
0620 14:50:34.173511 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:34.173513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:34.173522 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:50:34.173580 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:34.173581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:34.173629 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:34.173701 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:34.173878 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:34.174385 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:34.175169 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:34.175381 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:34.175447 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:34.175585 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:34.175687 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:34.175738 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:34.175756 (+ 18us) server_negotiation.cc:299] Negotiation successful
0620 14:50:34.175771 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260620 14:50:34.179875 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:34.177558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56020 (local address 127.0.98.60:44867)
0620 14:50:34.177672 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:34.177674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:34.177682 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:50:34.177720 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:34.177721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:34.177756 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:34.177823 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:34.177937 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:34.178436 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:34.179125 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:34.179325 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:34.179380 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:34.179538 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:34.179642 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:34.179703 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:34.179757 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 14:50:34.179778 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
I20260620 14:50:34.183832 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:34.181362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56026 (local address 127.0.98.60:44867)
0620 14:50:34.181403 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:34.181405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:34.181505 (+ 100us) server_negotiation.cc:407] Connection header received
0620 14:50:34.181601 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:34.181602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:34.181611 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:34.181636 (+ 25us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:34.181804 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:34.182390 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:34.183183 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:34.183436 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:34.183488 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:34.183610 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:34.183639 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:34.183685 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:34.183735 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 14:50:34.183754 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:50:34.184468 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:34.181289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.0.98.62:37889)
0620 14:50:34.181403 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:34.181405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:34.181557 (+ 152us) server_negotiation.cc:407] Connection header received
0620 14:50:34.181609 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:34.181610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:34.181663 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:34.181743 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:34.181906 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:34.182386 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:34.183473 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:34.183764 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:34.183820 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:34.184132 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:34.184244 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:34.184302 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:34.184365 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 14:50:34.184389 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":60,"threads_started":1}
I20260620 14:50:34.184928 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:34.181436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.0.98.61:39905)
0620 14:50:34.181476 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:34.181478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:34.181553 (+ 75us) server_negotiation.cc:407] Connection header received
0620 14:50:34.181612 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:34.181612 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:34.181624 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:34.181682 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:34.182514 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:34.183227 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:34.184269 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:34.184545 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:34.184604 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:34.184717 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:34.184752 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:34.184800 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:34.184860 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 14:50:34.184878 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:50:34.186085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 22588
I20260620 14:50:34.190364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 22663
I20260620 14:50:34.194543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 22738
2026-06-20T14:50:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2670 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:34 dist-test-slave-cx89 krb5kdc[22955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:34 dist-test-slave-cx89 krb5kdc[22955](info): set up 2 sockets
Jun 20 14:50:34 dist-test-slave-cx89 krb5kdc[22955](info): commencing operation
krb5kdc: starting...
W20260620 14:50:35.147284 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/test-user.keytab.
Jun 20 14:50:35 dist-test-slave-cx89 krb5kdc[22955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:50:35.253988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36513
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36513,127.0.98.61:42259,127.0.98.60:40069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:35.330385 22968 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:50:35.330583 22968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:35.330598 22968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:35.330618 22968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:35.331806 22968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:35.331856 22968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:35.331871 22968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:35.331882 22968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:35.331893 22968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:35.333367 22968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36513,127.0.98.61:42259,127.0.98.60:40069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:35.333585 22968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:35.333807 22968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:35.338608 22976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:35.338629 22973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:35.338621 22968 server_base.cc:1034] running on GCE node
W20260620 14:50:35.338608 22974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:35.339059 22968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:35.339350 22968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:35.340487 22968 hybrid_clock.cc:648] HybridClock initialized: now 1781967035340471 us; error 34 us; skew 500 ppm
Jun 20 14:50:35 dist-test-slave-cx89 krb5kdc[22955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:35.345904 22968 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:50:35.346333 22968 webserver.cc:460] Webserver started at http://127.0.98.62:34027/ using document root <none> and password file <none>
I20260620 14:50:35.346541 22968 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:35.346596 22968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:35.346701 22968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:35.347754 22968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-0/data/instance:
uuid: "26c9dd26c94242e1aa0b104da3493bb1"
format_stamp: "Formatted at 2026-06-20 14:50:35 on dist-test-slave-cx89"
server_key: "f94d98b1514ef66a05a5ab306d6f51a1"
server_key_iv: "b54422de9d6193812684eecf6e6b5f19"
server_key_version: "encryptionkey@0"
I20260620 14:50:35.348063 22968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-0/wal/instance:
uuid: "26c9dd26c94242e1aa0b104da3493bb1"
format_stamp: "Formatted at 2026-06-20 14:50:35 on dist-test-slave-cx89"
server_key: "f94d98b1514ef66a05a5ab306d6f51a1"
server_key_iv: "b54422de9d6193812684eecf6e6b5f19"
server_key_version: "encryptionkey@0"
I20260620 14:50:35.349241 22968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:50:35.349932 22983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:35.350083 22968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:35.350158 22968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-0/wal
uuid: "26c9dd26c94242e1aa0b104da3493bb1"
format_stamp: "Formatted at 2026-06-20 14:50:35 on dist-test-slave-cx89"
server_key: "f94d98b1514ef66a05a5ab306d6f51a1"
server_key_iv: "b54422de9d6193812684eecf6e6b5f19"
server_key_version: "encryptionkey@0"
I20260620 14:50:35.350222 22968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:35.369436 22968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:35.371582 22968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:35.371711 22968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:35.375393 22968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36513
I20260620 14:50:35.375463 23035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36513 every 8 connection(s)
I20260620 14:50:35.375788 22968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:35.376721 23036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:35.378329 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 22968
I20260620 14:50:35.378412 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-0/wal/instance
I20260620 14:50:35.378270 23036 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:35.378619 392 external_mini_cluster.cc:1428] Setting key d367b29b7b64dc402f8f811a47457b8b
Jun 20 14:50:35 dist-test-slave-cx89 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260620 14:50:35.387392 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.378918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36513 (local address 127.0.0.1:44676)
0620 14:50:35.379082 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.379126 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:35.379731 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:35.380032 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:35.380035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:35.380278 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:35.380433 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:35.380437 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.381527 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.381528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:35.382199 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:35.382201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.382670 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.382671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:35.382732 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.383187 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:35.383195 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:35.384494 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:35.386066 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:35.386072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:35.386074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:35.386241 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:35.386440 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:35.386441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:35.386441 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:35.386517 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:35.386864 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:35.386868 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:35.386931 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 14:50:35.387010 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:50:35.387228 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":86,"thread_start_us":78,"threads_started":1}
I20260620 14:50:35.387399 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.378964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44676 (local address 127.0.98.62:36513)
0620 14:50:35.379100 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.379110 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.379212 (+ 102us) server_negotiation.cc:407] Connection header received
0620 14:50:35.379885 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.379888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.379911 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.380013 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.380513 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.381457 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.382277 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.382586 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.382683 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.384566 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:35.384573 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:35.384573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:35.384580 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:35.385991 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.386306 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.386309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.386309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.386387 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.386567 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.386568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.386568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.386774 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:35.386883 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:35.387037 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 14:50:35.387156 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":53,"threads_started":1}
I20260620 14:50:35.388185 23036 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260620 14:50:35.388474 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.388384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42259 (local address 127.0.0.1:57398)
0620 14:50:35.388416 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.388455 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260620 14:50:35.388597 22984 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42259: connect: Connection refused (error 111)
I20260620 14:50:35.388954 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.388871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42259 (local address 127.0.0.1:57402)
0620 14:50:35.388908 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.388931 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 14:50:35.389039 23036 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42259: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42259: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 14:50:35.402467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42259
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36513,127.0.98.61:42259,127.0.98.60:40069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:50:35.438267 23036 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } attempt: 1
I20260620 14:50:35.438838 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.438669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42259 (local address 127.0.0.1:57404)
0620 14:50:35.438757 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.438805 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 14:50:35.439222 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.439145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42259 (local address 127.0.0.1:57406)
0620 14:50:35.439179 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.439204 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 14:50:35.439299 23036 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42259: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42259: connect: Connection refused (error 111)
W20260620 14:50:35.480849 23043 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:50:35.481083 23043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:35.481099 23043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:35.481117 23043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:35.482182 23043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:35.482229 23043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:35.482245 23043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:35.482256 23043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:35.482272 23043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:35.483692 23043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36513,127.0.98.61:42259,127.0.98.60:40069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:35.483872 23043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:35.484038 23043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:50:35.488479 23043 server_base.cc:1034] running on GCE node
W20260620 14:50:35.488554 23049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:35.488481 23051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:35.488611 23048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:35.488898 23043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:35.489167 23043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:35.490298 23043 hybrid_clock.cc:648] HybridClock initialized: now 1781967035490287 us; error 28 us; skew 500 ppm
Jun 20 14:50:35 dist-test-slave-cx89 krb5kdc[22955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:35.492795 23043 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 14:50:35.493235 23043 webserver.cc:460] Webserver started at http://127.0.98.61:33059/ using document root <none> and password file <none>
I20260620 14:50:35.493444 23043 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:35.493491 23043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:35.493614 23043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:35.494663 23043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-1/data/instance:
uuid: "d101de6c39f342deb34f69c6d75fb0ee"
format_stamp: "Formatted at 2026-06-20 14:50:35 on dist-test-slave-cx89"
server_key: "925d468b43042f08f8b2feabed5c65d2"
server_key_iv: "a1c0312aa9d92f1a2564a8132902a64d"
server_key_version: "encryptionkey@0"
I20260620 14:50:35.494984 23043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-1/wal/instance:
uuid: "d101de6c39f342deb34f69c6d75fb0ee"
format_stamp: "Formatted at 2026-06-20 14:50:35 on dist-test-slave-cx89"
server_key: "925d468b43042f08f8b2feabed5c65d2"
server_key_iv: "a1c0312aa9d92f1a2564a8132902a64d"
server_key_version: "encryptionkey@0"
I20260620 14:50:35.496245 23043 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:50:35.496946 23058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:35.497097 23043 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:35.497179 23043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-1/wal
uuid: "d101de6c39f342deb34f69c6d75fb0ee"
format_stamp: "Formatted at 2026-06-20 14:50:35 on dist-test-slave-cx89"
server_key: "925d468b43042f08f8b2feabed5c65d2"
server_key_iv: "a1c0312aa9d92f1a2564a8132902a64d"
server_key_version: "encryptionkey@0"
I20260620 14:50:35.497236 23043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:35.507438 23036 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } attempt: 2
I20260620 14:50:35.507884 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.507769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42259 (local address 127.0.0.1:57414)
0620 14:50:35.507821 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.507857 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 14:50:35.508160 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.508094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42259 (local address 127.0.0.1:57428)
0620 14:50:35.508124 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.508145 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 14:50:35.508232 23036 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42259: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42259: connect: Connection refused (error 111)
I20260620 14:50:35.516727 23043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:35.518527 23043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:35.518630 23043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:35.522164 23043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42259
I20260620 14:50:35.522203 23110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42259 every 8 connection(s)
I20260620 14:50:35.522502 23043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:50:35.523387 23111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:35.524923 23111 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:35.526826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 23043
I20260620 14:50:35.526947 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-1/wal/instance
I20260620 14:50:35.527149 392 external_mini_cluster.cc:1428] Setting key b8776ca1692e0522d298d481c7764ff8
Jun 20 14:50:35 dist-test-slave-cx89 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 14:50:35.534045 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.525544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44678 (local address 127.0.98.62:36513)
0620 14:50:35.525598 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.525600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.525765 (+ 165us) server_negotiation.cc:407] Connection header received
0620 14:50:35.526220 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.526221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.526236 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.526286 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.526988 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.527535 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.528769 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.529071 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.529132 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.531214 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:35.531215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:35.531216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:35.531222 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:35.532718 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.533088 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.533089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.533089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.533153 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.533352 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.533353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.533353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.533486 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:35.533566 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:35.533852 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 14:50:35.533939 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:50:35.534164 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.525526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36513 (local address 127.0.0.1:44678)
0620 14:50:35.525660 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.525693 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:35.526131 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:35.526349 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:35.526352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:35.526590 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:35.526866 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:35.526872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.527617 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.527618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:35.528697 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:35.528699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.529149 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.529150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:35.529250 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.529703 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:35.529711 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:35.531126 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:35.532791 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:35.532797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:35.532799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:35.533016 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:35.533212 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:35.533213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:35.533213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:35.533296 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:35.533579 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:35.533586 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:35.533720 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 14:50:35.533814 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 14:50:35.533982 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260620 14:50:35.534690 23111 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 14:50:35 dist-test-slave-cx89 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:50:35.541622 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.534953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42259 (local address 127.0.0.1:57438)
0620 14:50:35.535021 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.535032 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:35.535069 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:35.535258 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:35.535259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:35.535382 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:35.535467 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:35.535469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.536329 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.536330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:35.537119 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:35.537123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.537575 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.537576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:35.537647 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.538063 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:35.538067 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:35.539125 (+ 1058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:35.540731 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:35.540732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:35.540732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:35.540879 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:35.541046 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:35.541046 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:35.541047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:35.541096 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:35.541329 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:35.541330 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:35.541365 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:50:35.541424 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:50:35.541520 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 14:50:35.541695 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.535004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57438 (local address 127.0.98.61:42259)
0620 14:50:35.535143 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.535147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.535162 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:50:35.535176 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.535178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.535188 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.535249 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.535532 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.536257 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.537200 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.537493 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.537554 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.539196 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:35.539204 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:35.539205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:35.539211 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:35.540649 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.540945 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.540948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.540948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.540995 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.541143 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.541144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.541144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.541273 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:35.541341 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:35.541456 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 14:50:35.541586 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260620 14:50:35.542194 23111 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:35.542519 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.542418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40069 (local address 127.0.0.1:57620)
0620 14:50:35.542455 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.542497 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 14:50:35.542622 23061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40069: connect: Connection refused (error 111)
I20260620 14:50:35.542972 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.542889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40069 (local address 127.0.0.1:57632)
0620 14:50:35.542926 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.542950 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 14:50:35.543071 23111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40069: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40069: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 14:50:35.549882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskV7d6BE/build/release/bin/kudu
/tmp/dist-test-taskV7d6BE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40069
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36513,127.0.98.61:42259,127.0.98.60:40069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:50:35.592315 23111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } attempt: 1
I20260620 14:50:35.592823 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.592689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40069 (local address 127.0.0.1:57646)
0620 14:50:35.592743 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.592789 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 14:50:35.593221 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.593129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40069 (local address 127.0.0.1:57652)
0620 14:50:35.593170 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.593197 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:50:35.593312 23111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40069: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40069: connect: Connection refused (error 111)
I20260620 14:50:35.599355 23036 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } attempt: 3
Jun 20 14:50:35 dist-test-slave-cx89 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:50:35.606104 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.599777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57442 (local address 127.0.98.61:42259)
0620 14:50:35.599838 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.599840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.599851 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:50:35.599906 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.599907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.599920 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.599969 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.600292 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.600840 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.601568 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.601792 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.601854 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.603856 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:35.603858 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:35.603858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:35.603865 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:35.605124 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.605431 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.605431 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.605432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.605483 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.605645 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.605646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.605647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.605754 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:35.605808 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:35.605937 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 14:50:35.606007 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:50:35.606105 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.599724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42259 (local address 127.0.0.1:57442)
0620 14:50:35.599778 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.599790 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:35.599842 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:35.599981 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:35.599982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:35.600147 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:35.600215 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:35.600217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.600953 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.600954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:35.601487 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:35.601490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.601859 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.601861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:35.601937 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.602403 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:35.602407 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:35.603774 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:35.605212 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:35.605213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:35.605214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:35.605370 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:35.605545 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:35.605546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:35.605547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:35.605597 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:35.605806 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:35.605807 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:35.605852 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:50:35.605910 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:50:35.606006 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 14:50:35.606446 23036 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:35.606750 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.606661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40069 (local address 127.0.0.1:57658)
0620 14:50:35.606701 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.606726 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 14:50:35.607077 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.606983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40069 (local address 127.0.0.1:57672)
0620 14:50:35.607023 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.607055 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 14:50:35.607162 23036 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40069: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40069: connect: Connection refused (error 111)
W20260620 14:50:35.628218 23118 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:50:35.628420 23118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:35.628436 23118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:35.628454 23118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:35.629518 23118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:35.629549 23118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:35.629560 23118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:35.629571 23118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:35.629585 23118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:35.630950 23118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36513,127.0.98.61:42259,127.0.98.60:40069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:35.631127 23118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:35.631291 23118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:35.635957 23124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:35.635942 23123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:35.635927 23126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:35.636008 23118 server_base.cc:1034] running on GCE node
I20260620 14:50:35.636345 23118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:35.636606 23118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:35.637740 23118 hybrid_clock.cc:648] HybridClock initialized: now 1781967035637730 us; error 27 us; skew 500 ppm
I20260620 14:50:35.638302 23036 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } attempt: 1
I20260620 14:50:35.638650 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.638541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40069 (local address 127.0.0.1:57682)
0620 14:50:35.638584 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.638615 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 14:50:35.638955 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.638888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40069 (local address 127.0.0.1:57698)
0620 14:50:35.638921 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.638941 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 14:50:35.639037 23036 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40069: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40069: connect: Connection refused (error 111)
Jun 20 14:50:35 dist-test-slave-cx89 krb5kdc[22955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:35.640242 23118 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 14:50:35.640647 23118 webserver.cc:460] Webserver started at http://127.0.98.60:40669/ using document root <none> and password file <none>
I20260620 14:50:35.640836 23118 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:35.640880 23118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:35.640995 23118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:35.641970 23118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-2/data/instance:
uuid: "d9b601552aa4491e9262ffaa94533db9"
format_stamp: "Formatted at 2026-06-20 14:50:35 on dist-test-slave-cx89"
server_key: "ec662dce8a635a48ab28b606edeafd7c"
server_key_iv: "d9c27e687e7acf626e2254ac8ed719e0"
server_key_version: "encryptionkey@0"
I20260620 14:50:35.642293 23118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-2/wal/instance:
uuid: "d9b601552aa4491e9262ffaa94533db9"
format_stamp: "Formatted at 2026-06-20 14:50:35 on dist-test-slave-cx89"
server_key: "ec662dce8a635a48ab28b606edeafd7c"
server_key_iv: "d9c27e687e7acf626e2254ac8ed719e0"
server_key_version: "encryptionkey@0"
I20260620 14:50:35.643581 23118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:50:35.644332 23133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:35.644483 23118 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:35.644570 23118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-2/wal
uuid: "d9b601552aa4491e9262ffaa94533db9"
format_stamp: "Formatted at 2026-06-20 14:50:35 on dist-test-slave-cx89"
server_key: "ec662dce8a635a48ab28b606edeafd7c"
server_key_iv: "d9c27e687e7acf626e2254ac8ed719e0"
server_key_version: "encryptionkey@0"
I20260620 14:50:35.644642 23118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:35.661458 23111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } attempt: 2
I20260620 14:50:35.661897 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.661753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40069 (local address 127.0.0.1:57706)
0620 14:50:35.661811 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.661858 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 14:50:35.662245 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.662151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40069 (local address 127.0.0.1:57718)
0620 14:50:35.662195 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.662221 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 14:50:35.662326 23111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40069: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40069: connect: Connection refused (error 111)
I20260620 14:50:35.673209 23118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:35.674963 23118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:35.675091 23118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:35.678700 23118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40069
I20260620 14:50:35.678731 23185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40069 every 8 connection(s)
I20260620 14:50:35.679047 23118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:50:35.679949 23186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:35.681396 23186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:35.684149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskV7d6BE/build/release/bin/kudu as pid 23118
I20260620 14:50:35.684245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskV7d6BE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966748446763-392-0/minicluster-data/master-2/wal/instance
I20260620 14:50:35.684420 392 external_mini_cluster.cc:1428] Setting key c64c07e4a049706281029c2cc7c0d756
Jun 20 14:50:35 dist-test-slave-cx89 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 14:50:35 dist-test-slave-cx89 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:50:35.689667 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.682028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44692 (local address 127.0.98.62:36513)
0620 14:50:35.682093 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.682095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.682308 (+ 213us) server_negotiation.cc:407] Connection header received
0620 14:50:35.682756 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.682757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.682765 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.682817 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.683360 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.683852 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.684932 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.685205 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.685276 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.687216 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:35.687218 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:35.687218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:35.687225 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:35.688452 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.688803 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.688805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.688805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.688858 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.689079 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.689080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.689080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.689209 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:35.689270 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:35.689531 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 14:50:35.689601 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:50:35.689851 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.682008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36513 (local address 127.0.0.1:44692)
0620 14:50:35.682213 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.682251 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:35.682682 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:35.682873 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:35.682876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:35.683110 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:35.683299 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:35.683303 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.683911 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.683912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:35.684835 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:35.684838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.685286 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.685286 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:35.685391 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.685831 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:35.685839 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:35.687145 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:35.688523 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:35.688530 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:35.688533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:35.688740 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:35.688910 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:35.688911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:35.688911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:35.689025 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:35.689265 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:35.689271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:35.689389 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 14:50:35.689477 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:50:35.689628 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20260620 14:50:35.690348 23186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:35.692373 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.684732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57732 (local address 127.0.98.60:40069)
0620 14:50:35.684877 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.684882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.684900 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:50:35.684956 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.684958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.684976 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.685037 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.685632 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.686300 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.687036 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.687391 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.687490 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.689523 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:35.689528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:35.689528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:35.689534 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:35.691278 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.691657 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.691659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.691659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.691706 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.691869 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.691870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.691870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.692024 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:35.692159 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:35.692200 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 14:50:35.692271 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
Jun 20 14:50:35 dist-test-slave-cx89 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:50:35.696764 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.690737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57452 (local address 127.0.98.61:42259)
0620 14:50:35.690783 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.690784 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.690793 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:50:35.690813 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.690814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.690821 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.690864 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.691130 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.691764 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.692609 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.692877 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.692935 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.694547 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:35.694548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:35.694548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:35.694554 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:35.695762 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.696104 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.696105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.696105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.696160 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.696335 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.696335 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.696335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.696440 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:35.696488 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:35.696619 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 14:50:35.696679 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:50:35.696782 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.690663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42259 (local address 127.0.0.1:57452)
0620 14:50:35.690711 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.690722 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:35.690763 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:35.690868 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:35.690868 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:35.691006 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:35.691066 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:35.691067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.691833 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.691834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:35.692536 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:35.692539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.692939 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.692939 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:35.693003 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.693336 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:35.693340 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:35.694468 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:35.695817 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:35.695818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:35.695819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:35.696004 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:35.696223 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:35.696224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:35.696224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:35.696284 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:35.696490 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:35.696491 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:35.696532 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:50:35.696602 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:50:35.696687 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 14:50:35.697117 23186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:50:35 dist-test-slave-cx89 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:50:35.700189 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.693117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44704 (local address 127.0.98.62:36513)
0620 14:50:35.693156 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.693158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.693209 (+ 51us) server_negotiation.cc:407] Connection header received
0620 14:50:35.693294 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.693295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.693304 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.693355 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.693784 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.694232 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.695047 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.695328 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.695393 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.697707 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:35.697708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:35.697708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:35.697715 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:35.699102 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.699468 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.699469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.699469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.699531 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.699702 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.699702 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.699703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.699822 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:35.699966 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:35.700026 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 14:50:35.700103 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Jun 20 14:50:35 dist-test-slave-cx89 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:50:35.703670 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.697348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40069 (local address 127.0.0.1:57734)
0620 14:50:35.697391 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.697402 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:35.697444 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:35.697548 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:35.697548 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:35.697663 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:35.697694 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:35.697695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.698400 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.698401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:35.699126 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:35.699128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.699848 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.699849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:35.699917 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.700253 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:35.700257 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:35.701374 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:35.702662 (+ 1288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:35.702663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:35.702663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:35.702803 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:35.702949 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:35.702949 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:35.702950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:35.702998 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:35.703367 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:35.703367 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:35.703402 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:50:35.703480 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:50:35.703568 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 14:50:35.703717 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.697377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.0.98.60:40069)
0620 14:50:35.697406 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.697408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.697431 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:50:35.697490 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.697491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.697499 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.697542 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.697751 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.698328 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.699495 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.699787 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.699850 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.701427 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:35.701428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:35.701429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:35.701435 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:35.702595 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.702855 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.702856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.702857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.702900 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.703217 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.703218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.703218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.703309 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:35.703363 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:35.703500 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 14:50:35.703613 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":15}
Jun 20 14:50:35 dist-test-slave-cx89 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:50:35.704931 23186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9: Bootstrap starting.
I20260620 14:50:35.705804 23186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:35.706068 23186 log.cc:824] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:35.706725 23186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9: No bootstrap required, opened a new log
I20260620 14:50:35.706945 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.700819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57464 (local address 127.0.98.61:42259)
0620 14:50:35.700856 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.700858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.700866 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:50:35.700884 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.700885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.700893 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.700936 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.701374 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.702030 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.702628 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.702824 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.702874 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.704762 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:35.704763 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:35.704764 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:35.704770 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:35.705994 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.706307 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.706308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.706308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.706364 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.706518 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.706518 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.706519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.706621 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:35.706766 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:35.706799 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 14:50:35.706862 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:50:35.708202 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:35.708578 23186 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c9dd26c94242e1aa0b104da3493bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } } peers { permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } } peers { permanent_uuid: "d9b601552aa4491e9262ffaa94533db9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } }
I20260620 14:50:35.708695 23186 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b601552aa4491e9262ffaa94533db9, State: Initialized, Role: FOLLOWER
I20260620 14:50:35.708760 23186 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c9dd26c94242e1aa0b104da3493bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } } peers { permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } } peers { permanent_uuid: "d9b601552aa4491e9262ffaa94533db9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } }
I20260620 14:50:35.709055 23186 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:35.709012 23192 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c9dd26c94242e1aa0b104da3493bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } } peers { permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } } peers { permanent_uuid: "d9b601552aa4491e9262ffaa94533db9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } } }
I20260620 14:50:35.709110 23192 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:35.709231 23010 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "26c9dd26c94242e1aa0b104da3493bb1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44704
W20260620 14:50:35.711481 23203 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:50:35.711539 23203 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:50:35.714164 23036 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } attempt: 2
I20260620 14:50:35.711733 23118 master_runner.cc:186] Error getting master registration for 127.0.98.62:36513: OK, instance_id { permanent_uuid: "26c9dd26c94242e1aa0b104da3493bb1" instance_seqno: 1781967035374304 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:50:35.714365 23118 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 20 14:50:35 dist-test-slave-cx89 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:50:35.721280 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.714480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.0.98.60:40069)
0620 14:50:35.714530 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.714532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.714543 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:50:35.714563 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.714564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.714573 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.714638 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.714908 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.715619 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.716571 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.716882 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.716949 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.718681 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:35.718683 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:35.718683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:35.718690 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:35.720228 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.720583 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.720584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.720584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.720640 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.720809 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.720810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.720810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.720939 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:35.721001 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:35.721122 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 14:50:35.721193 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:50:35.721282 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.714386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40069 (local address 127.0.0.1:57740)
0620 14:50:35.714466 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.714475 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:35.714521 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:35.714643 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:35.714644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:35.714803 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:35.714845 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:35.714847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.715688 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.715689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:35.716490 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:35.716492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.716946 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.716947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:35.716997 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.717367 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:35.717372 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:35.718603 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:35.720303 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:35.720304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:35.720305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:35.720518 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:35.720703 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:35.720704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:35.720704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:35.720759 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:35.720991 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:35.720992 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:35.721035 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:50:35.721097 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:50:35.721183 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 14:50:35.723208 23036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1: Bootstrap starting.
I20260620 14:50:35.723919 23036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:35.724213 23036 log.cc:824] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:35.725049 23036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1: No bootstrap required, opened a new log
I20260620 14:50:35.726378 23036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c9dd26c94242e1aa0b104da3493bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } } peers { permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } } peers { permanent_uuid: "d9b601552aa4491e9262ffaa94533db9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } }
I20260620 14:50:35.726518 23036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c9dd26c94242e1aa0b104da3493bb1, State: Initialized, Role: FOLLOWER
I20260620 14:50:35.726601 23036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c9dd26c94242e1aa0b104da3493bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } } peers { permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } } peers { permanent_uuid: "d9b601552aa4491e9262ffaa94533db9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } }
I20260620 14:50:35.726858 23206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c9dd26c94242e1aa0b104da3493bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } } peers { permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } } peers { permanent_uuid: "d9b601552aa4491e9262ffaa94533db9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } } }
I20260620 14:50:35.726922 23036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:35.727010 23206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:35.726948 23010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:50:35.727108 23010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:35.727808 23010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c9dd26c94242e1aa0b104da3493bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } } peers { permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } } peers { permanent_uuid: "d9b601552aa4491e9262ffaa94533db9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } }
I20260620 14:50:35.728040 23010 leader_election.cc:288] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [CANDIDATE]: Term 1 election: Requested vote from peers d101de6c39f342deb34f69c6d75fb0ee (127.0.98.61:42259), d9b601552aa4491e9262ffaa94533db9 (127.0.98.60:40069)
I20260620 14:50:35.728538 23160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26c9dd26c94242e1aa0b104da3493bb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d9b601552aa4491e9262ffaa94533db9"
I20260620 14:50:35.728657 23160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:35.729455 23160 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c9dd26c94242e1aa0b104da3493bb1 in term 1.
I20260620 14:50:35.729835 22986 leader_election.cc:302] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c9dd26c94242e1aa0b104da3493bb1, d9b601552aa4491e9262ffaa94533db9; no voters:
I20260620 14:50:35.729962 23206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:35.730077 23206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [term 1 LEADER]: Becoming Leader. State: Replica: 26c9dd26c94242e1aa0b104da3493bb1, State: Running, Role: LEADER
I20260620 14:50:35.730151 23206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c9dd26c94242e1aa0b104da3493bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } } peers { permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } } peers { permanent_uuid: "d9b601552aa4491e9262ffaa94533db9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } }
I20260620 14:50:35.730526 23206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26c9dd26c94242e1aa0b104da3493bb1. Latest consensus state: current_term: 1 leader_uuid: "26c9dd26c94242e1aa0b104da3493bb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c9dd26c94242e1aa0b104da3493bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } } peers { permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } } peers { permanent_uuid: "d9b601552aa4491e9262ffaa94533db9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } } }
I20260620 14:50:35.730597 23206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [sys.catalog]: This master's current role is: LEADER
W20260620 14:50:35.730616 23217 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:50:35.730649 23217 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:50:35.731010 22968 master_runner.cc:186] Error getting master registration for 127.0.98.61:42259: OK, instance_id { permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" instance_seqno: 1781967035521209 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:50:35.733264 22968 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:50:35.734333 23228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:35.734568 23228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:35.738051 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.729154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.0.98.62:36513)
0620 14:50:35.729190 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.729192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.729881 (+ 689us) server_negotiation.cc:407] Connection header received
0620 14:50:35.729913 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.729914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.729924 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.729988 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.730846 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.731614 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.732595 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.732880 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.732918 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.734928 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:35.734929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:35.734930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:35.734937 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:35.736460 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.736995 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.736997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.736997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.737066 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.737360 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.737361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.737361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.737515 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:35.737586 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:35.737901 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 14:50:35.737969 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:50:35.741081 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.730884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.0.98.61:42259)
0620 14:50:35.730924 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.730926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.733501 (+ 2575us) server_negotiation.cc:407] Connection header received
0620 14:50:35.733622 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.733624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.733635 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.733687 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.734303 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.735028 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.736023 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.736733 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.736774 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.738063 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:35.738065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:35.738065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:35.738072 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:35.739621 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.739980 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.739981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.739982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.740041 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.740225 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.740226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.740227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.740412 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:35.740561 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:35.740904 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 14:50:35.740988 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:50:35.741506 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.729715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.0.98.60:40069)
0620 14:50:35.729754 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.729756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.730792 (+ 1036us) server_negotiation.cc:407] Connection header received
0620 14:50:35.730879 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.730880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.730889 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.730960 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.731663 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.732322 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.734681 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.734956 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.734993 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.737804 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:35.737805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:35.737806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:35.737812 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:35.740160 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.740490 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.740491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.740491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.740548 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.740820 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.740821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.740821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.740963 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:35.741033 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:35.741334 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 14:50:35.741412 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:50:35.746244 23160 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9 [term 1 FOLLOWER]: Refusing update from remote peer 26c9dd26c94242e1aa0b104da3493bb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:35.746599 23225 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9b601552aa4491e9262ffaa94533db9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:35.748250 23192 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26c9dd26c94242e1aa0b104da3493bb1. Latest consensus state: current_term: 1 leader_uuid: "26c9dd26c94242e1aa0b104da3493bb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c9dd26c94242e1aa0b104da3493bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } } peers { permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } } peers { permanent_uuid: "d9b601552aa4491e9262ffaa94533db9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } } }
I20260620 14:50:35.748342 23192 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:35.752755 23206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26c9dd26c94242e1aa0b104da3493bb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c9dd26c94242e1aa0b104da3493bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } } peers { permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } } peers { permanent_uuid: "d9b601552aa4491e9262ffaa94533db9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } } }
I20260620 14:50:35.752846 23206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:35.752942 23192 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26c9dd26c94242e1aa0b104da3493bb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c9dd26c94242e1aa0b104da3493bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } } peers { permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } } peers { permanent_uuid: "d9b601552aa4491e9262ffaa94533db9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } } }
I20260620 14:50:35.753018 23192 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:35.753371 23230 mvcc.cc:205] Tried to move back new op lower bound from 7298936978414059520 to 7298936978351689728. Current Snapshot: MvccSnapshot[applied={T|T < 7298936978414059520}]
I20260620 14:50:35.753455 23111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } attempt: 3
Jun 20 14:50:35 dist-test-slave-cx89 krb5kdc[22955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:50:35.759783 23228 catalog_manager.cc:1347] Generated new cluster ID: 1ff871390211468984b5699fc33a28f7
I20260620 14:50:35.759836 23228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:35.762799 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.753718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40069 (local address 127.0.0.1:57758)
0620 14:50:35.753772 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:35.753785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:35.753832 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:35.753961 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:35.753962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:35.754133 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:35.754180 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:35.754182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.755055 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.755056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:35.755922 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:35.755925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.756388 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.756389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:35.756460 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.756998 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:35.757003 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:35.758482 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:35.761449 (+ 2967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:35.761450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:35.761451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:35.761681 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:35.761987 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:35.761988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:35.761989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:35.762061 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:35.762486 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:35.762487 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:35.762558 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 14:50:35.762604 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 14:50:35.762701 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 14:50:35.763616 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:35.753824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57758 (local address 127.0.98.60:40069)
0620 14:50:35.753858 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.753860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.753868 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:50:35.753888 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.753888 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.753897 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.753952 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.754247 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.754982 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.756009 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:35.756311 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:35.756383 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:35.758565 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:35.758566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:35.758566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:35.758573 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:35.761298 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.761777 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.761778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.761811 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.761895 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:35.762140 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:35.762140 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:35.762141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:35.762389 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:35.763257 (+ 868us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:35.763327 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:50:35.763461 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:50:35.765323 23111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee: Bootstrap starting.
I20260620 14:50:35.766022 23111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:35.766299 23111 log.cc:824] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:35.774804 23228 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:35.776125 23228 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:35.776163 23111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee: No bootstrap required, opened a new log
I20260620 14:50:35.778165 23111 raft_consensus.cc:348] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c9dd26c94242e1aa0b104da3493bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } } peers { permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } } peers { permanent_uuid: "d9b601552aa4491e9262ffaa94533db9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } }
I20260620 14:50:35.778306 23111 raft_consensus.cc:729] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d101de6c39f342deb34f69c6d75fb0ee, State: Initialized, Role: FOLLOWER
I20260620 14:50:35.778381 23111 consensus_queue.cc:260] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c9dd26c94242e1aa0b104da3493bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } } peers { permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } } peers { permanent_uuid: "d9b601552aa4491e9262ffaa94533db9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } }
I20260620 14:50:35.778712 23111 sys_catalog.cc:564] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:35.779663 23084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:35.780546 23084 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee [term 1 FOLLOWER]: Refusing update from remote peer 26c9dd26c94242e1aa0b104da3493bb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:35.780885 23085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26c9dd26c94242e1aa0b104da3493bb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d101de6c39f342deb34f69c6d75fb0ee"
I20260620 14:50:35.780901 23206 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:35.781800 23085 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c9dd26c94242e1aa0b104da3493bb1 in term 1.
I20260620 14:50:35.783761 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26c9dd26c94242e1aa0b104da3493bb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c9dd26c94242e1aa0b104da3493bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } } peers { permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } } peers { permanent_uuid: "d9b601552aa4491e9262ffaa94533db9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } } }
I20260620 14:50:35.783915 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:35.784008 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26c9dd26c94242e1aa0b104da3493bb1. Latest consensus state: current_term: 1 leader_uuid: "26c9dd26c94242e1aa0b104da3493bb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c9dd26c94242e1aa0b104da3493bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } } peers { permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } } peers { permanent_uuid: "d9b601552aa4491e9262ffaa94533db9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } } }
I20260620 14:50:35.784102 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:35.784162 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26c9dd26c94242e1aa0b104da3493bb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c9dd26c94242e1aa0b104da3493bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } } peers { permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } } peers { permanent_uuid: "d9b601552aa4491e9262ffaa94533db9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } } }
I20260620 14:50:35.784248 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:35.785429 23252 catalog_manager.cc:1259] Loaded cluster ID: 1ff871390211468984b5699fc33a28f7
I20260620 14:50:35.785523 23252 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee: loading cluster ID for follower catalog manager: success
I20260620 14:50:35.786698 23252 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee: acquiring CA information for follower catalog manager: success
W20260620 14:50:35.786875 23252 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260620 14:50:35.793756 23206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26c9dd26c94242e1aa0b104da3493bb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c9dd26c94242e1aa0b104da3493bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36513 } } peers { permanent_uuid: "d101de6c39f342deb34f69c6d75fb0ee" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42259 } } peers { permanent_uuid: "d9b601552aa4491e9262ffaa94533db9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40069 } } }
I20260620 14:50:35.793883 23206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:35.797032 23228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26c9dd26c94242e1aa0b104da3493bb1: Generated new TSK 0
I20260620 14:50:35.797221 23228 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:50:35.822953 23039 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:35.821658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44722 (local address 127.0.98.62:36513)
0620 14:50:35.821705 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.821707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.822315 (+ 608us) server_negotiation.cc:407] Connection header received
0620 14:50:35.822417 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.822418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.822429 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.822459 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.822879 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44722: BlockingRecv error: recv got EOF from 127.0.0.1:44722 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 14:50:35.823696 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:35.822044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.0.98.60:40069)
0620 14:50:35.822083 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.822084 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.823195 (+ 1111us) server_negotiation.cc:407] Connection header received
0620 14:50:35.823283 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.823284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.823294 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.823320 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.823636 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57760: BlockingRecv error: recv got EOF from 127.0.0.1:57760 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 14:50:35.824710 23115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:35.824040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57482 (local address 127.0.98.61:42259)
0620 14:50:35.824075 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.824077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.824196 (+ 119us) server_negotiation.cc:407] Connection header received
0620 14:50:35.824280 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.824281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.824292 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.824318 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.824640 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57482: BlockingRecv error: recv got EOF from 127.0.0.1:57482 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 14:50:35.826934 23115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:35.826192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57498 (local address 127.0.98.61:42259)
0620 14:50:35.826224 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.826226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.826376 (+ 150us) server_negotiation.cc:407] Connection header received
0620 14:50:35.826473 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.826474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.826493 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.826521 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.826889 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57498: BlockingRecv error: recv got EOF from 127.0.0.1:57498 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 14:50:35.832510 23115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:35.831780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57504 (local address 127.0.98.61:42259)
0620 14:50:35.831814 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.831816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.832030 (+ 214us) server_negotiation.cc:407] Connection header received
0620 14:50:35.832118 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.832120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.832129 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.832158 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.832463 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57504: BlockingRecv error: recv got EOF from 127.0.0.1:57504 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 14:50:35.836397 23115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:35.835668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57510 (local address 127.0.98.61:42259)
0620 14:50:35.835701 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.835702 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.835899 (+ 197us) server_negotiation.cc:407] Connection header received
0620 14:50:35.836000 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.836001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.836011 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.836037 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.836347 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57510: BlockingRecv error: recv got EOF from 127.0.0.1:57510 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 14:50:35.842257 23115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:35.841518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57514 (local address 127.0.98.61:42259)
0620 14:50:35.841551 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.841553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.841748 (+ 195us) server_negotiation.cc:407] Connection header received
0620 14:50:35.841837 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.841838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.841848 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.841875 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.842207 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57514: BlockingRecv error: recv got EOF from 127.0.0.1:57514 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 14:50:35.852250 23115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:35.851398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.0.98.61:42259)
0620 14:50:35.851472 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.851474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.851706 (+ 232us) server_negotiation.cc:407] Connection header received
0620 14:50:35.851795 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.851796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.851807 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.851834 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.852195 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57520: BlockingRecv error: recv got EOF from 127.0.0.1:57520 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 14:50:35.870481 23115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:35.869575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57522 (local address 127.0.98.61:42259)
0620 14:50:35.869618 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.869620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.869877 (+ 257us) server_negotiation.cc:407] Connection header received
0620 14:50:35.869995 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.869996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.870008 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.870039 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.870421 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57522: BlockingRecv error: recv got EOF from 127.0.0.1:57522 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 14:50:35.904620 23115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:35.903863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57530 (local address 127.0.98.61:42259)
0620 14:50:35.903905 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.903907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.904079 (+ 172us) server_negotiation.cc:407] Connection header received
0620 14:50:35.904136 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.904137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.904150 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.904200 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.904557 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57530: BlockingRecv error: recv got EOF from 127.0.0.1:57530 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 14:50:35.970243 23115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:35.969603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57544 (local address 127.0.98.61:42259)
0620 14:50:35.969656 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:35.969658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:35.969704 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:50:35.969781 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:35.969783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:35.969795 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:35.969840 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:35.970178 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57544: BlockingRecv error: recv got EOF from 127.0.0.1:57544 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260620 14:50:36.099844 23115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:36.099224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.0.98.61:42259)
0620 14:50:36.099300 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:36.099302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:36.099346 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:50:36.099403 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:36.099404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:36.099436 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:36.099488 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:36.099783 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57548: BlockingRecv error: recv got EOF from 127.0.0.1:57548 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 14:50:36.357623 23115 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:36.357017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.0.98.61:42259)
0620 14:50:36.357070 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:36.357072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:36.357082 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:50:36.357128 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:36.357129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:36.357139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:36.357187 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:36.357562 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57552: BlockingRecv error: recv got EOF from 127.0.0.1:57552 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:50:36.711954 23203 catalog_manager.cc:1259] Loaded cluster ID: 1ff871390211468984b5699fc33a28f7
I20260620 14:50:36.712028 23203 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9: loading cluster ID for follower catalog manager: success
I20260620 14:50:36.712831 23203 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9: acquiring CA information for follower catalog manager: success
I20260620 14:50:36.713111 23203 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d9b601552aa4491e9262ffaa94533db9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:50:36.787501 23252 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d101de6c39f342deb34f69c6d75fb0ee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:50:36.873148 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:36.870659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57558 (local address 127.0.98.61:42259)
0620 14:50:36.870768 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:36.870770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:36.870796 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:50:36.870857 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:36.870857 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:36.870907 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:36.870974 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:36.871084 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:36.871601 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:36.872321 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:36.872583 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:36.872641 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:36.872799 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:36.872943 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:36.873014 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:36.873035 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 14:50:36.873058 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":53,"threads_started":1}
I20260620 14:50:36.876690 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:36.874685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57766 (local address 127.0.98.60:40069)
0620 14:50:36.874772 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:36.874774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:36.874781 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:50:36.874834 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:36.874835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:36.874867 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:36.874928 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:36.874988 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:36.875490 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:36.876127 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:36.876309 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:36.876348 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:36.876443 (+ 95us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:36.876533 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:36.876581 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:36.876596 (+ 15us) server_negotiation.cc:299] Negotiation successful
0620 14:50:36.876611 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70,"thread_start_us":42,"threads_started":1}
I20260620 14:50:36.880800 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:36.878119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.0.98.60:40069)
0620 14:50:36.878154 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:36.878156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:36.878356 (+ 200us) server_negotiation.cc:407] Connection header received
0620 14:50:36.878443 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:36.878444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:36.878453 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:36.878479 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:36.878639 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:36.879340 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:36.880102 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:36.880385 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:36.880443 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:36.880601 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:36.880631 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:36.880676 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:36.880716 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 14:50:36.880734 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:50:36.881075 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:36.878155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57560 (local address 127.0.98.61:42259)
0620 14:50:36.878204 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:36.878205 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:36.878256 (+ 51us) server_negotiation.cc:407] Connection header received
0620 14:50:36.878341 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:36.878342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:36.878353 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:36.878397 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:36.878485 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:36.879199 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:36.880197 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:36.880471 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:36.880519 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:36.880649 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:36.880688 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:36.880731 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:36.880813 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:50:36.880838 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:50:36.881078 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:36.878063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44730 (local address 127.0.98.62:36513)
0620 14:50:36.878218 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:36.878220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:36.878228 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:50:36.878288 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:36.878289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:36.878341 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:36.878416 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:36.878578 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:36.879141 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:36.880235 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:36.880435 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:36.880487 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:36.880681 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:36.880876 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:36.880935 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:36.880979 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 14:50:36.881000 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":51,"threads_started":1}
I20260620 14:50:36.882513 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 22968
I20260620 14:50:36.887284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 23043
I20260620 14:50:36.891731 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskV7d6BE/build/release/bin/kudu with pid 23118
2026-06-20T14:50:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2696 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5367 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (288449 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS